DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1: /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240503 07:59:09.721897 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.722083 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.725208 9030 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:10.160985 9030 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 07:59:10.553947 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server c167a0a433d34b74affed13cbf96e302
W20240503 07:59:10.554349 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:10.554427 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554497 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.554554 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554594 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.554672 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.554788 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc23875e4c9c4380bd7862acd33adc2b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554834 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 3d08a06d53b14381bee6223c4e1a6027 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:10.554929 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.554967 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555027 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555102 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6d629539a2874def9c039beeee9b9490: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555156 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555212 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555253 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555294 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555336 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555378 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555433 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555476 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555518 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555558 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555598 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555640 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555706 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555742 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555776 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6d629539a2874def9c039beeee9b9490: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555804 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:10.572374 9030 rebalancer_tool.cc:1074] tablet 0e9ce52b9ae543cca72fb92efafc3c4d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.585142 9030 rebalancer_tool.cc:1074] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.615864 9030 rebalancer_tool.cc:1074] tablet c597bc1a19274f55b9d60558db35fcae: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.637740 9030 rebalancer_tool.cc:1074] tablet 610160f6ba94483a9bdcf966c413e5d1: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.656585 9030 rebalancer_tool.cc:1074] tablet e874c9cd82864973a92fb743e45ccf67: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.671169 9030 rebalancer_tool.cc:1074] tablet 5538a709d8d440cb8f006a163f259389: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.685398 9030 rebalancer_tool.cc:1074] tablet c7e3555fb9214a40bd8dbf937b87cbaa: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.699667 9030 rebalancer_tool.cc:1074] tablet 065d5d0851e840939802896f9bddb84c: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.701303 9030 rebalancer_tool.cc:1084] tablet bc0992eb4c7a471f9646518dd20e7c26: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move ignored: Not found: bc0992eb4c7a471f9646518dd20e7c26: tablet not found
I20240503 07:59:10.701377 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:10.877220 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877312 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877343 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877378 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 3d08a06d53b14381bee6223c4e1a6027 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:10.877506 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc23875e4c9c4380bd7862acd33adc2b: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877537 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877590 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877629 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877681 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877712 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server c167a0a433d34b74affed13cbf96e302
W20240503 07:59:10.877741 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877781 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877812 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877841 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877873 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.877902 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.877931 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877985 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.878016 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.878062 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878091 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.878120 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878152 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.878192 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.878221 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878253 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:10.891927 9030 rebalancer_tool.cc:1074] tablet 09c32a1e047844a3b09f2c790d735c9b: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.905635 9030 rebalancer_tool.cc:1074] tablet 8384cbf3f368424d9f8a9060efe9c0f2: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.918112 9030 rebalancer_tool.cc:1074] tablet 93eb2db48ef449a486a0347a76a8a5ea: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.931584 9030 rebalancer_tool.cc:1074] tablet 82de209205b342cfaac10506fdac9303: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.958356 9030 rebalancer_tool.cc:1074] tablet 5710dff79184444797bb46d981ae6f55: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.974042 9030 rebalancer_tool.cc:1074] tablet 7f417c61594348428ee9514ded49ee88: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.000555 9030 rebalancer_tool.cc:1074] tablet 2f943ecfe59f432c861a54b16c62572f: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:11.057148 9030 rebalancer_tool.cc:1074] tablet 5d95cc5335534f02aa62b00fb8b9c33c: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:11.093199 9030 rebalancer_tool.cc:1074] tablet 39b01318dc1543cba2cc991e9db91c28: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:11.115651 9030 rebalancer_tool.cc:1074] tablet fd67fb07203e4e99919ec0317bdf077d: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.137940 9030 rebalancer_tool.cc:1074] tablet 5631f3e97fe04013a2a7140f44260003: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:11.158378 9030 rebalancer_tool.cc:1074] tablet d4308bc87be24f28b6566ce76cae6e50: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:11.189060 9030 rebalancer_tool.cc:1123] tablet e874c9cd82864973a92fb743e45ccf67: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: e874c9cd82864973a92fb743e45ccf67: tablet not found
I20240503 07:59:11.303802 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:11.577474 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.577666 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:11.577759 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.577996 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578090 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:11.578140 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578197 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578246 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578308 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578356 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578405 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578449 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.578495 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578562 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578608 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578685 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578749 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.578788 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:11.578874 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578919 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578963 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.579027 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.579088 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.579138 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.579186 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.579780 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.625955 9030 rebalancer_tool.cc:1074] tablet 5f05300c9002408084f0b47d5e47591d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.708299 9030 rebalancer_tool.cc:1123] tablet fd67fb07203e4e99919ec0317bdf077d: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: fd67fb07203e4e99919ec0317bdf077d: tablet not found
I20240503 07:59:11.735075 9030 rebalancer_tool.cc:1123] tablet 5710dff79184444797bb46d981ae6f55: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: 5710dff79184444797bb46d981ae6f55: tablet not found
I20240503 07:59:11.759567 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:12.220618 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220737 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220836 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220952 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:12.220991 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221032 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221069 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221123 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221297 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221478 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221534 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221599 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221642 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221684 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221741 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221783 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221894 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221966 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.222004 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.222044 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.222084 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.222126 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:12.239388 9030 rebalancer_tool.cc:1074] tablet 0e3f58a27f2a4a2593e8219b1d7bd894: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.257222 9030 rebalancer_tool.cc:1074] tablet 7d855e5edf05471983dfdbd40f37560f: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:12.295229 9030 rebalancer_tool.cc:1136] tablet 5538a709d8d440cb8f006a163f259389: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.304636 9030 rebalancer_tool.cc:1136] tablet 0e9ce52b9ae543cca72fb92efafc3c4d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.312577 9030 rebalancer_tool.cc:1136] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.394002 9030 rebalancer_tool.cc:1136] tablet 065d5d0851e840939802896f9bddb84c: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.398692 9030 rebalancer_tool.cc:1136] tablet 610160f6ba94483a9bdcf966c413e5d1: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.403508 9030 rebalancer_tool.cc:1136] tablet 8384cbf3f368424d9f8a9060efe9c0f2: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.413825 9030 rebalancer_tool.cc:1123] tablet 09c32a1e047844a3b09f2c790d735c9b: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: 09c32a1e047844a3b09f2c790d735c9b: tablet not found
I20240503 07:59:12.428738 9030 rebalancer_tool.cc:1136] tablet c597bc1a19274f55b9d60558db35fcae: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:12.446739 9030 rebalancer_tool.cc:1074] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.459218 9030 rebalancer_tool.cc:1074] tablet f5604af852814da58385f072eb9600aa: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.472692 9030 rebalancer_tool.cc:1074] tablet 11fbfb84cef74316a77b3ccfab49a19b: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.488082 9030 rebalancer_tool.cc:1074] tablet 9772b487ca864ad98e9d79d8435c34c6: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:12.507879 9030 rebalancer_tool.cc:1074] tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.520573 9030 rebalancer_tool.cc:1074] tablet 435f73fb9f384b10b353df19037bcac3: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.532786 9030 rebalancer_tool.cc:1074] tablet edf2c780d0d64db79dca88eac77d4478: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.686764 9030 rebalancer_tool.cc:1136] tablet 39b01318dc1543cba2cc991e9db91c28: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:12.692605 9030 rebalancer_tool.cc:1136] tablet 93eb2db48ef449a486a0347a76a8a5ea: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.714674 9030 rebalancer_tool.cc:1136] tablet 5631f3e97fe04013a2a7140f44260003: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.747504 9030 rebalancer_tool.cc:1136] tablet 82de209205b342cfaac10506fdac9303: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.782550 9030 rebalancer_tool.cc:1136] tablet 7f417c61594348428ee9514ded49ee88: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.789462 9030 rebalancer_tool.cc:1136] tablet 2f943ecfe59f432c861a54b16c62572f: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.798152 9030 rebalancer_tool.cc:1136] tablet 5d95cc5335534f02aa62b00fb8b9c33c: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.810915 9030 rebalancer_tool.cc:1074] tablet a2762b2e6de842bb8c898aea5b3910ba: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.828131 9030 rebalancer_tool.cc:1074] tablet 824c42be8152494fad2cbc76ece56981: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.829293 9030 rebalancer_tool.cc:1084] tablet dc995af1acb9459f9096867bc3b075f4: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move ignored: Not found: dc995af1acb9459f9096867bc3b075f4: tablet not found
I20240503 07:59:12.829353 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:13.115881 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.115970 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116029 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.116092 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116153 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116227 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116353 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.116434 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:13.116550 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.116612 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.116659 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:13.116703 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116757 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116917 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:13.145793 9030 rebalancer_tool.cc:1074] tablet b29dafc7b564493a8fdf064e7d3461ef: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.167183 9030 rebalancer_tool.cc:1074] tablet 8861b52c306e43d8b0615202cf7d8e35: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:13.195382 9030 rebalancer_tool.cc:1074] tablet a01fa99be75f498a8394aabbfabb3f5a: '9553fe3c4ccd4e91b7c176aad82fec09' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:13.211068 9030 rebalancer_tool.cc:1074] tablet 2f943ecfe59f432c861a54b16c62572f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.211911 9030 rebalancer_tool.cc:1084] tablet 18677aa766454b8c98a1a2f4a48feee2: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move ignored: Not found: 18677aa766454b8c98a1a2f4a48feee2: tablet not found
I20240503 07:59:13.211973 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:13.426029 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.426100 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.426265 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.426383 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426430 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.426486 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.426545 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.426663 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426784 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.426831 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426872 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.426940 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426985 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:13.443693 9030 rebalancer_tool.cc:1074] tablet 94d5216ff8654561b44a0cfd0f9b1625: 'c167a0a433d34b74affed13cbf96e302' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:13.460263 9030 rebalancer_tool.cc:1074] tablet f534eefb7c334889a9c5d02175ae2d94: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:13.489634 9030 rebalancer_tool.cc:1123] tablet b29dafc7b564493a8fdf064e7d3461ef: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: b29dafc7b564493a8fdf064e7d3461ef: tablet not found
I20240503 07:59:13.550467 9030 rebalancer_tool.cc:1123] tablet 5f05300c9002408084f0b47d5e47591d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 5f05300c9002408084f0b47d5e47591d: tablet not found
I20240503 07:59:13.556649 9030 rebalancer_tool.cc:1136] tablet d4308bc87be24f28b6566ce76cae6e50: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:13.557471 9030 rebalancer_tool.cc:1123] tablet 11fbfb84cef74316a77b3ccfab49a19b: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 11fbfb84cef74316a77b3ccfab49a19b: tablet not found
I20240503 07:59:13.609514 9030 rebalancer_tool.cc:1074] tablet b2e0f5369e8349eba7188b39ae6bac30: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:13.735255 9030 rebalancer_tool.cc:1136] tablet c7e3555fb9214a40bd8dbf937b87cbaa: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:13.736071 9030 rebalancer_tool.cc:1123] tablet 0e3f58a27f2a4a2593e8219b1d7bd894: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 0e3f58a27f2a4a2593e8219b1d7bd894: tablet not found
I20240503 07:59:13.757048 9030 rebalancer_tool.cc:1136] tablet 7d855e5edf05471983dfdbd40f37560f: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:13.801738 9030 rebalancer_tool.cc:1074] tablet 5631f3e97fe04013a2a7140f44260003: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.825026 9030 rebalancer_tool.cc:1074] tablet 60f643dce3eb4c669b36b0714a31651b: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.888356 9030 rebalancer_tool.cc:1136] tablet f5604af852814da58385f072eb9600aa: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:13.935043 9030 rebalancer_tool.cc:1074] tablet 39b01318dc1543cba2cc991e9db91c28: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:14.044137 9030 rebalancer_tool.cc:1136] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.074894 9030 rebalancer_tool.cc:1136] tablet edf2c780d0d64db79dca88eac77d4478: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:14.080552 9030 rebalancer_tool.cc:1136] tablet 9772b487ca864ad98e9d79d8435c34c6: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:14.102175 9030 rebalancer_tool.cc:1074] tablet bfd4259777e5492c904e6d65a36d92ed: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:14.122247 9030 rebalancer_tool.cc:1074] tablet f635e30fb6974fa1a79d8a46eecc3d5e: '3d08a06d53b14381bee6223c4e1a6027' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:14.215744 9030 rebalancer_tool.cc:1136] tablet 435f73fb9f384b10b353df19037bcac3: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:14.226025 9030 rebalancer_tool.cc:1136] tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.264636 9030 rebalancer_tool.cc:1074] tablet 2d3f03610866454595f4b6b87b7cd9cb: 'c167a0a433d34b74affed13cbf96e302' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:14.280819 9030 rebalancer_tool.cc:1074] tablet 5060df68b95e48a8a849e77ba926e608: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:14.397436 9030 rebalancer_tool.cc:1136] tablet 824c42be8152494fad2cbc76ece56981: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:14.456878 9030 rebalancer_tool.cc:1074] tablet f4e4ca6638764fb49f3afcab7688e2b3: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:14.791864 9030 rebalancer_tool.cc:1123] tablet b2e0f5369e8349eba7188b39ae6bac30: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move is abandoned: Not found: b2e0f5369e8349eba7188b39ae6bac30: tablet not found
I20240503 07:59:14.797763 9030 rebalancer_tool.cc:1136] tablet 2f943ecfe59f432c861a54b16c62572f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:14.804311 9030 rebalancer_tool.cc:1136] tablet 8861b52c306e43d8b0615202cf7d8e35: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.844938 9030 rebalancer_tool.cc:1136] tablet a2762b2e6de842bb8c898aea5b3910ba: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.871966 9030 rebalancer_tool.cc:1074] tablet 558c0eb4a4334dbe91ac60fad4e820dc: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:14.874138 9030 rebalancer_tool.cc:1084] tablet aa754582dc9e4518965f3a2c1836cbee: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move ignored: Not found: aa754582dc9e4518965f3a2c1836cbee: tablet not found
I20240503 07:59:14.874200 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:15.148578 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:15.148819 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 751d986bca094ff3b81b8f94513d43ec: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:15.148880 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:15.148983 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table af669f53d59f4eb4bdc4b498325400dd: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:15.170928 9030 rebalancer_tool.cc:1074] tablet c5b4c776eac24f739d85e09707cd751b: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:15.191627 9030 rebalancer_tool.cc:1074] tablet c8cf3d6a3ace46769854ed61f340fb25: '616fdda6ba714cb7be01a564de2b9db2' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:15.207793 9030 rebalancer_tool.cc:1074] tablet 551f25dcee034b7894498e3c748b0f0e: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:15.220139 9030 rebalancer_tool.cc:1074] tablet 974c815ff8e94190a7bf28444cb3cd9f: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:15.240937 9030 rebalancer_tool.cc:1074] tablet e103e0e9cb16416dba7f8bbfc08ceac8: '3d08a06d53b14381bee6223c4e1a6027' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:15.262666 9030 rebalancer_tool.cc:1074] tablet 82de209205b342cfaac10506fdac9303: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:15.355584 9030 rebalancer_tool.cc:1136] tablet f534eefb7c334889a9c5d02175ae2d94: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:15.359935 9030 rebalancer_tool.cc:1136] tablet a01fa99be75f498a8394aabbfabb3f5a: '9553fe3c4ccd4e91b7c176aad82fec09' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.364805 9030 rebalancer_tool.cc:1136] tablet 94d5216ff8654561b44a0cfd0f9b1625: 'c167a0a433d34b74affed13cbf96e302' -> 'a97073f89fd44702bccb74a9b76f858d' move completed: OK
I20240503 07:59:15.385699 9030 rebalancer_tool.cc:1074] tablet 09866b93f75545a19c9451556254da48: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:15.399915 9030 rebalancer_tool.cc:1074] tablet c597bc1a19274f55b9d60558db35fcae: 'a97073f89fd44702bccb74a9b76f858d' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:15.486433 9030 rebalancer_tool.cc:1136] tablet 60f643dce3eb4c669b36b0714a31651b: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:15.530176 9030 rebalancer_tool.cc:1074] tablet 8861b52c306e43d8b0615202cf7d8e35: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:15.634739 9030 rebalancer_tool.cc:1136] tablet 39b01318dc1543cba2cc991e9db91c28: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.678766 9030 rebalancer_tool.cc:1074] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:15.778195 9030 rebalancer_tool.cc:1136] tablet bfd4259777e5492c904e6d65a36d92ed: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:15.948845 9030 rebalancer_tool.cc:1136] tablet f635e30fb6974fa1a79d8a46eecc3d5e: '3d08a06d53b14381bee6223c4e1a6027' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:15.964808 9030 rebalancer_tool.cc:1136] tablet 2d3f03610866454595f4b6b87b7cd9cb: 'c167a0a433d34b74affed13cbf96e302' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.993288 9030 rebalancer_tool.cc:1074] tablet d4308bc87be24f28b6566ce76cae6e50: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:16.083006 9030 rebalancer_tool.cc:1136] tablet f4e4ca6638764fb49f3afcab7688e2b3: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.089006 9030 rebalancer_tool.cc:1136] tablet 5060df68b95e48a8a849e77ba926e608: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:16.137351 9030 rebalancer_tool.cc:1074] tablet afeff5ce33bc4205a8599bbb66b24672: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:16.163012 9030 rebalancer_tool.cc:1074] tablet a833fb4980c04673a335d020a3ad3dd3: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:16.567724 9030 rebalancer_tool.cc:1136] tablet 558c0eb4a4334dbe91ac60fad4e820dc: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:16.583222 9030 rebalancer_tool.cc:1136] tablet 5631f3e97fe04013a2a7140f44260003: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:16.673249 9030 rebalancer_tool.cc:1074] tablet 435f73fb9f384b10b353df19037bcac3: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:16.699066 9030 rebalancer_tool.cc:1074] tablet b61209b73f1f4872b5eccc823b1fa3df: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:16.781275 9030 rebalancer_tool.cc:1136] tablet c597bc1a19274f55b9d60558db35fcae: 'a97073f89fd44702bccb74a9b76f858d' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:16.804905 9030 rebalancer_tool.cc:1136] tablet 551f25dcee034b7894498e3c748b0f0e: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.811823 9030 rebalancer_tool.cc:1136] tablet c8cf3d6a3ace46769854ed61f340fb25: '616fdda6ba714cb7be01a564de2b9db2' -> 'c167a0a433d34b74affed13cbf96e302' move completed: OK
I20240503 07:59:16.820207 9030 rebalancer_tool.cc:1136] tablet c5b4c776eac24f739d85e09707cd751b: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:16.832350 9030 rebalancer_tool.cc:1136] tablet e103e0e9cb16416dba7f8bbfc08ceac8: '3d08a06d53b14381bee6223c4e1a6027' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.862488 9030 rebalancer_tool.cc:1074] tablet edf2c780d0d64db79dca88eac77d4478: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:16.882623 9030 rebalancer_tool.cc:1074] tablet afcd161598db41d7962caca61f63ef77: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:16.926805 9030 rebalancer_tool.cc:1136] tablet 974c815ff8e94190a7bf28444cb3cd9f: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:16.964197 9030 rebalancer_tool.cc:1136] tablet 82de209205b342cfaac10506fdac9303: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:16.993131 9030 rebalancer_tool.cc:1074] tablet 9772b487ca864ad98e9d79d8435c34c6: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:17.077139 9030 rebalancer_tool.cc:1136] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:17.676190 9030 rebalancer_tool.cc:1136] tablet afeff5ce33bc4205a8599bbb66b24672: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:17.704397 9030 rebalancer_tool.cc:1136] tablet d4308bc87be24f28b6566ce76cae6e50: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:17.742390 9030 rebalancer_tool.cc:1136] tablet a833fb4980c04673a335d020a3ad3dd3: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:18.305913 9030 rebalancer_tool.cc:1136] tablet b61209b73f1f4872b5eccc823b1fa3df: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:18.321419 9030 rebalancer_tool.cc:1136] tablet 435f73fb9f384b10b353df19037bcac3: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.357174 9030 rebalancer_tool.cc:1136] tablet edf2c780d0d64db79dca88eac77d4478: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:18.371577 9030 rebalancer_tool.cc:1136] tablet 9772b487ca864ad98e9d79d8435c34c6: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.623713 9030 rebalancer_tool.cc:1136] tablet afcd161598db41d7962caca61f63ef77: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.636446 9030 rebalancer_tool.cc:1136] tablet 09866b93f75545a19c9451556254da48: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:18.849134 9030 rebalancer_tool.cc:1136] tablet 8861b52c306e43d8b0615202cf7d8e35: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
W20240503 07:59:19.076227 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f467f94f8014d3e979398f9ea0857e3: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:19.076323 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 68cd74d1f907476d8cccfd32a1339623: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:19.096592 9030 rebalancer_tool.cc:1074] tablet 5b745ba777b2476fbae78706b35125f6: 'c8d71787690245e681c85a087086e7ed' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.112599 9030 rebalancer_tool.cc:1074] tablet edaf4aa5285f49c297ab412ceb179be9: '3d08a06d53b14381bee6223c4e1a6027' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.126675 9030 rebalancer_tool.cc:1074] tablet 60f643dce3eb4c669b36b0714a31651b: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:19.139557 9030 rebalancer_tool.cc:1074] tablet e1b2ea716ca64e309efccd291db503d6: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.154740 9030 rebalancer_tool.cc:1074] tablet b61209b73f1f4872b5eccc823b1fa3df: '616fdda6ba714cb7be01a564de2b9db2' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.167088 9030 rebalancer_tool.cc:1074] tablet f7442d12e0234db38d896c09a9e73162: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.192380 9030 rebalancer_tool.cc:1074] tablet cfd4deb6a5104141bfa51fe4114bb708: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.212666 9030 rebalancer_tool.cc:1074] tablet 551f25dcee034b7894498e3c748b0f0e: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.228022 9030 rebalancer_tool.cc:1074] tablet 3e889be4506e4f01a978f9e94f6b18d2: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:19.245142 9030 rebalancer_tool.cc:1074] tablet 00bcbd5440ff4d56b1975edc4041b428: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.258183 9030 rebalancer_tool.cc:1074] tablet fc7d26466129405bbe6db2721823bae1: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.274138 9030 rebalancer_tool.cc:1074] tablet fa8b855a6985404da6bd34daa0eb9ae7: '616fdda6ba714cb7be01a564de2b9db2' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.287663 9030 rebalancer_tool.cc:1074] tablet 2054e7790d7745c2a1b08e903c64d1b7: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:19.300256 9030 rebalancer_tool.cc:1074] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
W20240503 07:59:19.934450 13800 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:19.739648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:40925 (local address 127.0.0.1:34174)
0503 07:59:19.739916 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:19.739930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 07:59:19.740007 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 07:59:19.933892 (+193885us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20240503 07:59:19.934860 9030 rebalancer_tool.cc:1123] tablet 00bcbd5440ff4d56b1975edc4041b428: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 07:59:19.955201 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): unacceptable health status UNAVAILABLE
I20240503 07:59:20.194370 6298 raft_consensus.cc:1203] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:20.195957 12930 consensus_queue.cc:1035] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.199775 13924 consensus_queue.cc:1035] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.211159 6690 raft_consensus.cc:1203] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 07:59:20.217242 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:20.221431 13872 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.221855 13381 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.225920 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47] alter complete (version 3)
I20240503 07:59:20.286144 6430 raft_consensus.cc:1203] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.293671 13175 consensus_queue.cc:1035] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.294214 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.296891 9347 consensus_queue.cc:1035] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.305990 6235 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.324342 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:20.329777 13018 raft_consensus.cc:1050] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.329859 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.330111 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.330183 13018 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.330453 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.330687 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.331580 8842 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.332046 6690 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.332415 12608 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.332545 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.332702 12608 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.332820 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.333070 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6db695ea2a48c49e687d95df2af58d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.333298 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6db695ea2a48c49e687d95df2af58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335070 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.335263 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335286 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.335476 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335592 13933 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:20.336270 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6db695ea2a48c49e687d95df2af58d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.336483 13933 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.336489 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6db695ea2a48c49e687d95df2af58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.337293 8976 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.337785 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.338042 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.338182 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.338280 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.339882 13938 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:20.340415 13934 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.341400 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet dbd4400894d14f839670d209468f1929
I20240503 07:59:20.341555 13940 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:20.342224 6690 raft_consensus.cc:1203] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.338757 13932 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.342401 13934 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.342414 13940 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.343366 13938 tablet_bootstrap.cc:654] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.344312 13934 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.344457 13934 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:20.344707 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd4400894d14f839670d209468f1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.345022 13934 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.345141 13934 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.345172 13934 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.345273 13934 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.345364 6151 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.345580 13934 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.345688 13938 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:20.345763 13938 ts_tablet_manager.cc:1367] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:20.345793 13934 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.346940 13934 tablet_bootstrap.cc:654] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.347182 13938 raft_consensus.cc:348] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.347308 13938 raft_consensus.cc:374] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.347371 13938 raft_consensus.cc:729] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.347497 13938 consensus_queue.cc:260] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.347965 13933 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:20.348047 13933 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 07:59:20.348310 13938 ts_tablet_manager.cc:1398] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.348453 13933 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.348531 13933 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.348558 13933 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.348587 13931 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.348660 13933 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.348809 8842 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.348946 13933 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.349093 13933 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:20.350208 13933 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.350293 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd4400894d14f839670d209468f1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.350524 13946 tablet_replica.cc:330] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.350557 6560 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.350630 13946 raft_consensus.cc:2229] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:20.350809 13946 raft_consensus.cc:2258] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.352063 8842 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.352074 6691 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.352434 13018 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.352579 13931 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.353041 13946 ts_tablet_manager.cc:1875] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.353053 8976 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.353289 6691 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.354890 13946 ts_tablet_manager.cc:1888] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.354961 13946 log.cc:1196] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbd4400894d14f839670d209468f1929
I20240503 07:59:20.355191 13946 ts_tablet_manager.cc:1909] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:20.355448 13940 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:20.355525 13940 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 07:59:20.355852 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47]) successfully deleted
I20240503 07:59:20.356029 13940 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.356185 13940 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.356254 13940 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.356400 13940 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.356945 13018 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.357333 13940 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.357839 13933 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:20.357892 13933 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:20.358237 13933 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.358320 13933 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.358366 13933 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.358458 13933 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.358686 13933 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.358626 8842 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.360541 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edaf4aa5285f49c297ab412ceb179be9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.360965 6155 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.363124 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet edaf4aa5285f49c297ab412ceb179be9 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:20.363369 6690 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.365797 13944 tablet_replica.cc:330] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.366036 13944 raft_consensus.cc:2229] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.366226 13944 raft_consensus.cc:2258] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.366912 13944 ts_tablet_manager.cc:1875] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.367380 13937 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:20.367580 13934 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.367650 13934 ts_tablet_manager.cc:1367] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240503 07:59:20.368002 13934 raft_consensus.cc:348] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.368075 13934 raft_consensus.cc:374] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.368095 13934 raft_consensus.cc:729] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.368146 13937 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.368191 13934 consensus_queue.cc:260] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.368471 13934 ts_tablet_manager.cc:1398] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.368975 13949 tablet_replica.cc:330] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:20.369252 13937 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:20.369338 13937 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.369750 13937 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.369877 13937 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.369932 13937 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.370080 13937 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.370385 13937 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.370559 13937 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:20.370682 13944 ts_tablet_manager.cc:1888] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.370772 13944 log.cc:1196] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbd4400894d14f839670d209468f1929
I20240503 07:59:20.369123 13949 raft_consensus.cc:2229] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.371003 13949 raft_consensus.cc:2258] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.371038 13944 ts_tablet_manager.cc:1909] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:20.371366 13937 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.371481 13949 ts_tablet_manager.cc:1875] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.371806 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47]) successfully deleted
I20240503 07:59:20.373229 13937 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:20.373301 13937 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.373664 13937 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.373791 13937 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.373833 13937 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.373945 13937 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.374164 13937 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.374256 13949 ts_tablet_manager.cc:1888] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.374327 13949 log.cc:1196] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/edaf4aa5285f49c297ab412ceb179be9
I20240503 07:59:20.374900 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet edaf4aa5285f49c297ab412ceb179be9 (table rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]) successfully deleted
I20240503 07:59:20.379088 6560 raft_consensus.cc:1203] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.434762 13478 raft_consensus.cc:1050] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.435117 13478 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.436311 6296 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.436849 6430 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.436895 8575 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.437242 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.437463 13478 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.437633 12241 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.440061 13478 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.440352 6296 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.441825 6430 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.442483 8575 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.442780 6150 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.445134 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:20.446187 8842 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.447207 6430 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.447564 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.447764 6296 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.448927 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.450551 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.450567 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.450767 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.450789 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.451437 11508 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.452710 13940 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:20.453051 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.453086 6430 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.453255 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.453716 13940 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.454921 13940 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:20.455000 13940 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:20.454968 6156 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.455319 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b745ba777b2476fbae78706b35125f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.455411 13940 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.455504 13940 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.455528 13940 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.455619 13940 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.455848 13940 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.456022 13934 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.456861 13934 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.459112 6296 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.459249 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b745ba777b2476fbae78706b35125f6 with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.459962 13944 tablet_replica.cc:330] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.460081 13944 raft_consensus.cc:2229] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.460191 13944 raft_consensus.cc:2258] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.460227 13957 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:20.461330 13934 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.461408 13934 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:20.461819 13934 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.461923 13934 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.461962 13934 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.462072 13934 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.462162 13957 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.462365 13934 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.463126 13957 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:20.463204 13957 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.463586 13957 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.463682 13957 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.463711 13957 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.463827 13957 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.464115 13957 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.465276 13944 ts_tablet_manager.cc:1875] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.466995 13944 ts_tablet_manager.cc:1888] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.467056 13944 log.cc:1196] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5b745ba777b2476fbae78706b35125f6
I20240503 07:59:20.467489 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 5b745ba777b2476fbae78706b35125f6 (table rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]) successfully deleted
I20240503 07:59:20.502321 13180 raft_consensus.cc:1050] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.502652 13180 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.503777 8842 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.503865 8575 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.504230 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.504706 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.505007 6430 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.505391 13924 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.506706 13180 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.507134 8575 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.508638 8842 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.509641 6430 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.511090 6151 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.515919 8976 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.516875 8842 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.516907 6430 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.517211 12930 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.517400 13924 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.518800 13180 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.519253 6430 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.519253 8842 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.520179 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e1b2ea716ca64e309efccd291db503d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.521622 6156 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.523594 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b2ea716ca64e309efccd291db503d6 with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.523761 13944 tablet_replica.cc:330] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.523897 13944 raft_consensus.cc:2229] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.524039 13944 raft_consensus.cc:2258] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.524587 13944 ts_tablet_manager.cc:1875] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.526121 13944 ts_tablet_manager.cc:1888] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.526182 13944 log.cc:1196] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e1b2ea716ca64e309efccd291db503d6
I20240503 07:59:20.526561 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet e1b2ea716ca64e309efccd291db503d6 (table rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]) successfully deleted
I20240503 07:59:20.557751 9347 raft_consensus.cc:1050] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:20.558122 9347 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.559278 6560 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559363 6430 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559533 8842 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559859 13969 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.560066 13175 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.560237 9347 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.562860 13175 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.563524 13953 raft_consensus.cc:1050] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.563387 6560 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.563907 13953 consensus_queue.cc:237] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.564797 6430 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.566638 8842 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.566803 6155 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:59:20.568413 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.569042 6430 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.569497 12241 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.570585 6690 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.571182 6560 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.571561 12241 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.571753 6428 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.572259 6559 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.572535 13278 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.572952 13278 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.573176 11508 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.573779 13924 raft_consensus.cc:482] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.573881 13924 raft_consensus.cc:504] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.574262 13924 leader_election.cc:288] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.574182 6559 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.575327 6428 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
W20240503 07:59:20.575553 8913 leader_election.cc:334] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.575577 13969 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.577349 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6db695ea2a48c49e687d95df2af58d" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:20.577457 6690 raft_consensus.cc:2454] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.577519 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e889be4506e4f01a978f9e94f6b18d2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.577940 8912 leader_election.cc:302] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.578177 13924 raft_consensus.cc:2781] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.578231 13924 raft_consensus.cc:482] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.578266 13924 raft_consensus.cc:3037] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.578924 6151 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.579200 13924 raft_consensus.cc:504] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.579521 13924 leader_election.cc:288] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.579802 6156 catalog_manager.cc:5484] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.580447 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6db695ea2a48c49e687d95df2af58d" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:20.580543 6690 raft_consensus.cc:3037] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.581406 6690 raft_consensus.cc:2454] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
W20240503 07:59:20.581733 8913 leader_election.cc:334] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.582398 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e889be4506e4f01a978f9e94f6b18d2 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.582432 8912 leader_election.cc:302] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.582573 13946 tablet_replica.cc:330] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.582473 6430 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.582691 13946 raft_consensus.cc:2229] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.582806 13946 raft_consensus.cc:2258] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.582782 6559 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.583699 13946 ts_tablet_manager.cc:1875] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.584306 12434 raft_consensus.cc:482] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.584393 12434 raft_consensus.cc:504] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.584636 13924 raft_consensus.cc:2781] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.584707 13924 raft_consensus.cc:686] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.584793 12434 leader_election.cc:288] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.584841 13924 consensus_queue.cc:237] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.586316 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:20.586436 8575 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:20.586658 6363 leader_election.cc:302] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.586779 13946 ts_tablet_manager.cc:1888] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.586823 12434 raft_consensus.cc:2781] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.586864 12434 raft_consensus.cc:482] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.586856 13946 log.cc:1196] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3e889be4506e4f01a978f9e94f6b18d2
I20240503 07:59:20.586897 12434 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.587584 6151 catalog_manager.cc:5484] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.587814 12434 raft_consensus.cc:504] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.589607 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 3e889be4506e4f01a978f9e94f6b18d2 (table rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]) successfully deleted
I20240503 07:59:20.589951 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:20.590058 8842 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:20.590971 12434 leader_election.cc:288] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.591313 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:20.591410 8842 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.592053 8842 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:20.592119 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:20.592221 8575 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.592933 6365 leader_election.cc:302] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.593142 8575 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:20.593152 12434 raft_consensus.cc:2781] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.593230 12434 raft_consensus.cc:686] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:20.593353 12434 consensus_queue.cc:237] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.596401 6155 catalog_manager.cc:5484] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.688905 12344 raft_consensus.cc:1050] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.689313 12344 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.690642 6690 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.690685 6296 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.691016 8976 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.691051 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.691236 13931 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.691416 12344 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.693271 12344 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.694906 6296 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.695106 6690 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.695760 8976 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.696202 6154 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.697816 13953 raft_consensus.cc:1050] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.698227 13953 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.699641 6430 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.699788 8975 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.700366 11508 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.700573 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.700747 6559 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.701951 6298 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:20.702157 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.702311 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.702453 6690 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.703161 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.704679 12608 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.705197 6690 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.706224 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7442d12e0234db38d896c09a9e73162 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.707425 6298 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.707638 6151 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.710460 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet f7442d12e0234db38d896c09a9e73162 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:20.710639 13949 tablet_replica.cc:330] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:20.710759 13949 raft_consensus.cc:2229] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.710891 13949 raft_consensus.cc:2258] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.711412 13949 ts_tablet_manager.cc:1875] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.711786 11508 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.712185 6430 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.713438 13949 ts_tablet_manager.cc:1888] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.713511 13949 log.cc:1196] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f7442d12e0234db38d896c09a9e73162
I20240503 07:59:20.713527 8975 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.714766 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet f7442d12e0234db38d896c09a9e73162 (table rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]) successfully deleted
I20240503 07:59:20.715384 6154 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.717758 8842 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.718735 6430 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.718845 8975 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.719154 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.719332 11508 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.720535 13478 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.721140 6430 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.722402 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc7d26466129405bbe6db2721823bae1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.722788 8975 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
W20240503 07:59:20.724363 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.724944 6155 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.731709 11508 raft_consensus.cc:482] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.731782 11508 raft_consensus.cc:504] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.732064 11508 leader_election.cc:288] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.732290 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:20.732384 6430 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:20.732378 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:20.732482 8975 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:20.732666 8779 leader_election.cc:302] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.732861 11508 raft_consensus.cc:2781] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.732918 11508 raft_consensus.cc:482] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.732944 11508 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.733557 11508 raft_consensus.cc:504] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.733826 11508 leader_election.cc:288] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.734059 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:20.734153 6430 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.734298 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:20.734400 8975 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.734916 6430 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:20.735265 8975 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:20.735242 8779 leader_election.cc:302] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.735502 11508 raft_consensus.cc:2781] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.735589 11508 raft_consensus.cc:686] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:20.735693 11508 consensus_queue.cc:237] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.737213 6155 catalog_manager.cc:5484] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.741977 11524 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.742059 11524 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:20.742349 11524 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:20.742694 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:20.743294 8913 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.743371 8913 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:20.743486 11524 raft_consensus.cc:2726] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.771991 13248 raft_consensus.cc:1050] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.772346 13248 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.773669 6298 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.773728 8976 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.774058 13114 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.774154 6559 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.774284 13989 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.774533 13248 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.776130 13114 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.776551 8976 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.776528 6559 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.779863 6155 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.780011 6298 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.785701 11524 raft_consensus.cc:482] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.785775 11524 raft_consensus.cc:504] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.786214 11524 leader_election.cc:288] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:20.786530 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:20.786571 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:20.786657 8575 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.786659 6559 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.786952 8910 leader_election.cc:302] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:20.787107 11524 raft_consensus.cc:2781] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.787160 11524 raft_consensus.cc:482] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.787185 11524 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.787981 13927 raft_consensus.cc:1050] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.787978 11524 raft_consensus.cc:504] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.788236 11524 leader_election.cc:288] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:20.788368 13927 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.788605 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:20.788695 6559 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.789575 6559 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.789598 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:20.789690 8575 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.789834 6690 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:20.790241 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:20.790225 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:20.790460 8575 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.790611 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:20.791088 8910 leader_election.cc:302] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:20.791489 11524 raft_consensus.cc:2781] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.791563 11524 raft_consensus.cc:686] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.791688 11524 consensus_queue.cc:237] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
W20240503 07:59:20.793618 6235 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.793823 6155 catalog_manager.cc:5484] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.794114 13927 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.795707 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.796370 6690 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.797303 6155 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.800063 6298 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.801074 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:20.801110 6690 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:20.801430 13537 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:20.801560 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:20.803009 13927 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.803488 6690 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.803580 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.804426 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551f25dcee034b7894498e3c748b0f0e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.805763 6154 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.829605 13924 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.829686 13924 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.830013 13924 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.830281 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
W20240503 07:59:20.831126 8913 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.831203 8913 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302
I20240503 07:59:20.831327 13924 raft_consensus.cc:2726] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.869983 13927 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.870079 13927 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:20.870412 13927 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:20.870725 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:20.871343 6235 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.871412 6235 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:20.871526 13927 raft_consensus.cc:2726] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.898016 12241 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.898119 12241 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.898483 12241 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.898789 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
W20240503 07:59:20.899399 8779 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.899472 8779 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2; no voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.899595 12241 raft_consensus.cc:2726] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.901401 13924 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.901476 13924 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.901738 13924 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.902022 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:20.902135 8842 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.902398 8912 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:20.902539 13924 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:20.902577 13924 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
W20240503 07:59:20.902583 8913 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.902630 13924 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:20.903455 13924 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.903729 13924 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.903985 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:20.904080 8842 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:20.904909 8913 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.905005 8842 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 2.
I20240503 07:59:20.905283 8912 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.905438 13924 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:20.905537 13924 raft_consensus.cc:686] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.905661 13924 consensus_queue.cc:237] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.907416 13961 raft_consensus.cc:1050] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.907418 13931 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.907503 13931 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.907785 13961 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.907841 13931 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.908463 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
W20240503 07:59:20.908823 6497 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.908799 6151 catalog_manager.cc:5484] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.909051 6496 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a; no voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.909209 6430 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.909253 13931 raft_consensus.cc:2726] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.909530 6690 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.909891 13922 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.910182 13922 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.910250 8842 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.910559 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.912223 13922 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.912853 8842 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.913064 13278 raft_consensus.cc:1050] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:20.913400 13278 consensus_queue.cc:237] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.914009 6430 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.914420 6690 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.914544 6559 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.915017 13969 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 07:59:20.915385 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.916620 8976 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.917002 13175 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.917230 6151 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.918489 13969 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.920024 8575 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.920230 6559 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.920262 8976 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.921972 6430 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.922493 6690 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.922859 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.923035 6154 catalog_manager.cc:5484] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.924599 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.924958 12035 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.926461 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa8b855a6985404da6bd34daa0eb9ae7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.926913 6690 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.927008 6430 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.927420 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:20.929700 6156 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.930646 6559 raft_consensus.cc:1261] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.930752 8575 raft_consensus.cc:1261] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.931006 13924 consensus_queue.cc:1035] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.931244 13924 consensus_queue.cc:1035] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.932912 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8b855a6985404da6bd34daa0eb9ae7 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.933099 13946 tablet_replica.cc:330] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.933223 13946 raft_consensus.cc:2229] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.933337 13946 raft_consensus.cc:2258] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.933940 13946 ts_tablet_manager.cc:1875] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.935243 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.935534 14012 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:20.936138 13946 ts_tablet_manager.cc:1888] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.936213 13946 log.cc:1196] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/fa8b855a6985404da6bd34daa0eb9ae7
I20240503 07:59:20.936728 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet fa8b855a6985404da6bd34daa0eb9ae7 (table rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]) successfully deleted
I20240503 07:59:20.937454 6151 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.938338 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.939792 14013 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:20.939714 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.940440 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.940490 14014 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.941284 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.941386 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 1)
W20240503 07:59:20.945673 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fc7d26466129405bbe6db2721823bae1: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.978050 12930 consensus_queue.cc:1035] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.981268 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.994494 13999 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.994580 13999 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.994890 13999 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:20.995206 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
W20240503 07:59:20.995909 8517 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.995981 8517 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302
I20240503 07:59:20.996090 13999 raft_consensus.cc:2726] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.037695 12241 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.037802 12241 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.038144 12241 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:21.038590 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
W20240503 07:59:21.039026 8779 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.039093 8779 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:21.039198 12241 raft_consensus.cc:2726] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.045714 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:21.050981 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.051134 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.051265 14014 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:21.051818 14012 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.051966 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.052238 14013 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.057188 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 2)
I20240503 07:59:21.058388 13989 consensus_queue.cc:1035] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.061854 12434 consensus_queue.cc:1035] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.070039 13999 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.070134 13999 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.070643 13999 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:21.070969 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:21.071100 6298 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
W20240503 07:59:21.071451 8517 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.071538 8514 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.071683 13999 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.071743 13999 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.071794 13999 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.072598 13999 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.072947 13999 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:21.073240 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:21.073343 6298 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.073772 8517 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.074239 6298 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.074530 8514 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.074687 13999 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.074787 13999 raft_consensus.cc:686] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.074936 13999 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.076658 6156 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:21.081722 13018 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.081808 13018 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.082123 13018 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.082468 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:21.083017 6497 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.083072 6497 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:21.083176 13018 raft_consensus.cc:2726] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.107744 13999 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.107829 13999 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.108155 13999 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.108433 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:21.108556 6559 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.108816 8514 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.108989 13999 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.109041 13999 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.109077 13999 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.109153 8517 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.109843 13999 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.110270 13999 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.110412 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:21.110517 6559 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.110992 8517 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.111217 6559 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.111431 8514 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.111565 13999 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.111644 13999 raft_consensus.cc:686] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.111766 13999 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.113495 6156 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.150990 12434 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.151103 12434 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.151455 12434 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.151723 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:21.151861 8842 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:21.152105 6365 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:21.152350 12434 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.152427 12434 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.152470 12434 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.152621 6366 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.153277 12434 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.153576 12434 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.153829 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:21.153975 8843 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.154484 6366 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.154873 8843 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 2.
I20240503 07:59:21.155133 6365 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.155299 12434 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.155402 12434 raft_consensus.cc:686] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:21.155529 12434 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.157403 6156 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.162391 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:21.164655 13927 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.164752 13927 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.165184 13927 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.165781 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.165892 8975 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:21.166375 6234 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:21.166649 13927 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.166699 13927 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.166733 13927 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.166934 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.167603 13927 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.167901 13927 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.168177 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.168277 8975 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.168841 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.169121 8975 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 2.
W20240503 07:59:21.169867 6235 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.170441 6235 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.171401 6234 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.171775 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.172427 13927 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.172518 13927 raft_consensus.cc:686] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:21.172649 13927 consensus_queue.cc:237] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.172966 13979 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.173041 13979 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.173383 13979 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.174604 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.174746 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:21.175062 6232 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.175226 13537 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.175274 13537 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.175315 13537 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.176043 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 3)
I20240503 07:59:21.176296 13537 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.176620 13537 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.177735 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.177840 8575 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.178584 6235 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.178699 6235 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.178709 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 2.
I20240503 07:59:21.178702 6156 catalog_manager.cc:5484] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.179065 6232 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.179255 13537 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.179368 13537 raft_consensus.cc:686] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:21.179497 13537 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.181833 6156 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:21.187541 13999 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.187582 11524 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.187618 13999 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.187673 11524 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.187942 13999 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.188004 11524 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.188196 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.188297 8975 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.188658 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:21.188802 6559 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:21.188824 8516 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
W20240503 07:59:21.188954 8517 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.189067 8910 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:21.189123 13999 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.189167 13999 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
W20240503 07:59:21.189155 8913 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.189200 13999 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.189306 11524 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.189353 11524 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.189378 11524 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.189872 13999 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.190069 11524 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.190410 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.190495 11524 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.190506 8975 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.190860 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:21.190968 6559 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.190999 13999 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
W20240503 07:59:21.191109 8517 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.191367 8975 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.191607 8516 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.191684 6559 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 2.
I20240503 07:59:21.191752 13999 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.191825 13999 raft_consensus.cc:686] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.191975 13999 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.192214 8910 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:21.192364 11524 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 07:59:21.192443 8913 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.192463 11524 raft_consensus.cc:686] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:21.192570 11524 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.194847 6156 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.195057 6151 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.209089 12241 consensus_queue.cc:1035] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.211644 11508 consensus_queue.cc:1035] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.278317 6151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.278470 6151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.280220 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.280828 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:21.280943 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:21.281117 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:21.281198 14033 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:21.281232 14034 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:21.281344 14033 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.281355 14035 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:21.281388 14034 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:21.281451 14033 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.281459 14035 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.281553 14035 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.281607 14034 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.282076 14033 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282115 14035 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282114 14034 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282452 8843 raft_consensus.cc:1261] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:21.283514 11524 consensus_queue.cc:1035] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:21.284332 14034 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.284384 14033 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.284417 14034 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.284440 14033 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.284660 14034 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:21.284754 14033 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:21.285362 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
I20240503 07:59:21.285779 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
I20240503 07:59:21.285948 14035 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.286021 14035 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.286571 14035 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:21.287268 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
W20240503 07:59:21.287348 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.309939 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:21.311094 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.315935 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316205 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316327 8561 tablet_service.cc:1467] Processing CreateTablet for tablet ef76600ded194f318cf97f157845bea8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316368 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316584 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef76600ded194f318cf97f157845bea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316687 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316808 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.317026 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.317080 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.317214 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.319504 8957 tablet_service.cc:1467] Processing CreateTablet for tablet ef76600ded194f318cf97f157845bea8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.319742 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef76600ded194f318cf97f157845bea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.321230 14039 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.322088 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.322207 14038 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:21.322347 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.322839 14039 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.323117 14038 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.323891 14039 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.323987 14039 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:21.324425 14039 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.324520 14039 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.324537 14042 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:21.324553 14039 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.324658 14039 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.322088 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.324945 14039 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.324973 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.325107 14039 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.325623 14042 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.325877 14039 tablet_bootstrap.cc:654] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.326552 14042 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:21.326627 14042 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:21.326956 14039 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.327029 14039 ts_tablet_manager.cc:1367] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.327031 14042 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.327103 14042 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.327128 14042 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.327234 14042 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.327440 14039 raft_consensus.cc:348] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.327529 14039 raft_consensus.cc:374] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.327580 14039 raft_consensus.cc:729] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.327739 14039 consensus_queue.cc:260] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.328016 14039 ts_tablet_manager.cc:1398] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.328554 14038 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:21.328645 14038 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:21.329097 14038 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.329205 14038 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.329281 14038 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.329432 14038 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.329706 14038 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.329897 14042 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:21.330044 14042 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:21.330840 14042 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.331823 14042 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:21.331918 14042 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:21.332355 14042 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.332475 14042 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.332517 14042 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.332631 14042 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.332926 14042 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.333130 14037 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.334491 14037 tablet_bootstrap.cc:654] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.337101 14037 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.337368 14037 ts_tablet_manager.cc:1367] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:21.337751 14037 raft_consensus.cc:348] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.337837 14037 raft_consensus.cc:374] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.337867 14037 raft_consensus.cc:729] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.337980 14037 consensus_queue.cc:260] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.340926 14037 ts_tablet_manager.cc:1398] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:21.341084 14037 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.341756 14037 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.342875 14037 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.342937 14037 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.343247 14037 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.343323 14037 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.343343 14037 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.343458 14037 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.343685 14037 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:21.344992 14037 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.345890 14037 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.346846 14037 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.346920 14037 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.347294 14037 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.347371 14037 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.347399 14037 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.347501 14037 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.347945 14037 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.363312 11508 raft_consensus.cc:482] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.363404 11508 raft_consensus.cc:504] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.363751 11508 leader_election.cc:288] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:21.364040 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.364059 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:21.364133 8575 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:21.364169 6690 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:21.364373 8776 leader_election.cc:302] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.364538 11508 raft_consensus.cc:2781] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.364590 11508 raft_consensus.cc:482] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.364625 11508 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.365329 11508 raft_consensus.cc:504] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.365609 11508 leader_election.cc:288] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:21.365856 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.365926 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:21.365958 8575 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.366024 6690 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.366664 6690 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:21.366739 8575 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:21.366911 8778 leader_election.cc:302] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:21.367266 11508 raft_consensus.cc:2781] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.367344 11508 raft_consensus.cc:686] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:21.367458 11508 consensus_queue.cc:237] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.369303 6151 catalog_manager.cc:5484] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.376536 13999 raft_consensus.cc:482] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.376610 13999 raft_consensus.cc:504] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.376916 13999 leader_election.cc:288] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.377213 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef76600ded194f318cf97f157845bea8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.377344 8975 raft_consensus.cc:2454] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.377755 8516 leader_election.cc:302] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.377913 13999 raft_consensus.cc:2781] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.377961 13999 raft_consensus.cc:482] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.377995 13999 raft_consensus.cc:3037] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.377993 8517 leader_election.cc:334] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.378691 13999 raft_consensus.cc:504] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.379004 13999 leader_election.cc:288] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.379640 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef76600ded194f318cf97f157845bea8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.379750 8975 raft_consensus.cc:3037] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.379940 8517 leader_election.cc:334] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.380473 8975 raft_consensus.cc:2454] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.380745 8516 leader_election.cc:302] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.380946 13999 raft_consensus.cc:2781] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.381037 13999 raft_consensus.cc:686] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.381153 13999 consensus_queue.cc:237] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.381598 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:21.383132 6154 catalog_manager.cc:5484] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.386955 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 07d78407a3354848826ab84e5b750ce0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]), partition=
I20240503 07:59:21.387185 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d78407a3354848826ab84e5b750ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.388835 14037 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.389144 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 07d78407a3354848826ab84e5b750ce0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]), partition=
I20240503 07:59:21.389351 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d78407a3354848826ab84e5b750ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.389604 14037 tablet_bootstrap.cc:654] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.390616 14037 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.390735 14037 ts_tablet_manager.cc:1367] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:21.391162 14039 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.391178 14037 raft_consensus.cc:348] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.391291 14037 raft_consensus.cc:374] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.391324 14037 raft_consensus.cc:729] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.391467 14037 consensus_queue.cc:260] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.391750 14037 ts_tablet_manager.cc:1398] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.392012 14039 tablet_bootstrap.cc:654] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.393143 14039 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.393235 14039 ts_tablet_manager.cc:1367] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.393633 14039 raft_consensus.cc:348] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.393714 14039 raft_consensus.cc:374] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.393735 14039 raft_consensus.cc:729] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.393867 14039 consensus_queue.cc:260] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.394117 14039 ts_tablet_manager.cc:1398] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.517561 6298 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.517930 13999 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:59:21.520320 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.570469 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.582266 6559 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.583628 13922 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.588639 13924 raft_consensus.cc:482] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.588757 13924 raft_consensus.cc:504] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.589196 13924 leader_election.cc:288] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.589437 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d78407a3354848826ab84e5b750ce0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.589558 8575 raft_consensus.cc:2454] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:21.589790 8910 leader_election.cc:302] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.589927 13924 raft_consensus.cc:2781] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.589993 13924 raft_consensus.cc:482] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.590036 13924 raft_consensus.cc:3037] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.590153 8913 leader_election.cc:334] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.590934 13924 raft_consensus.cc:504] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.591219 13924 leader_election.cc:288] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.591456 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d78407a3354848826ab84e5b750ce0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.591567 8575 raft_consensus.cc:3037] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.592420 8575 raft_consensus.cc:2454] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:21.592674 8910 leader_election.cc:302] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.592869 13924 raft_consensus.cc:2781] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.592976 13924 raft_consensus.cc:686] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:21.593102 13924 consensus_queue.cc:237] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
W20240503 07:59:21.593295 8913 leader_election.cc:334] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.594835 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.601714 6154 catalog_manager.cc:5484] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.630599 14056 raft_consensus.cc:482] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.630698 14056 raft_consensus.cc:504] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.631099 14056 leader_election.cc:288] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:21.631387 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.631400 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:21.631490 8975 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.631510 8843 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.631839 8516 leader_election.cc:302] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.632083 14056 raft_consensus.cc:2781] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.632154 14056 raft_consensus.cc:482] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.632200 14056 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.633081 14056 raft_consensus.cc:504] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.633407 14056 leader_election.cc:288] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:21.633641 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.633730 8975 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.633714 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:21.633816 8843 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.634433 8975 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.634621 8843 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.634682 8516 leader_election.cc:302] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.634848 14056 raft_consensus.cc:2781] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.634945 14056 raft_consensus.cc:686] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.635144 14056 consensus_queue.cc:237] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.637208 6154 catalog_manager.cc:5484] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.706758 8843 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:21.707135 13989 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:59:21.709969 6365 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b61209b73f1f4872b5eccc823b1fa3df. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.711203 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.711352 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.715363 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.719358 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:21.720635 8975 raft_consensus.cc:1261] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:21.721194 13537 consensus_queue.cc:1035] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:21.723016 8575 raft_consensus.cc:1261] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:21.723534 12930 consensus_queue.cc:1035] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:21.723927 8913 consensus_peers.cc:485] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.725504 8575 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.726090 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.727475 6151 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.727980 14063 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.728140 6559 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.728307 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:21.728596 12930 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.728680 14065 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.730031 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.732908 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 07:59:21.733187 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.733196 14062 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.737560 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 1)
W20240503 07:59:21.740212 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.774189 6298 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Advancing to term 2
I20240503 07:59:21.775300 6298 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:21.775647 14056 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.777141 14056 raft_consensus.cc:1050] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
W20240503 07:59:21.777212 14056 raft_consensus.cc:1053] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Unable to promote non-voter c167a0a433d34b74affed13cbf96e302: Illegal state: Leader has not yet committed an operation in its own term
I20240503 07:59:21.781292 8975 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:21.781610 14056 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.784348 14056 raft_consensus.cc:1050] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:21.784726 14056 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.786801 6298 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 07:59:21.786800 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.787143 8975 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:21.787261 13922 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:21.787760 14070 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:21.789089 13999 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.789633 6298 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.791306 8975 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.792114 6151 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:59:21.833552 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.840432 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:21.844887 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.845268 14062 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.845544 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.845858 14065 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.849850 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 2)
I20240503 07:59:21.868474 13953 consensus_queue.cc:1035] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.871040 12241 consensus_queue.cc:1035] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.917565 14056 consensus_queue.cc:1035] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:21.944645 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:21.952306 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:21.957231 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.957047 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.961932 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 3)
W20240503 07:59:21.968989 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): CreateTablet RPC failed for tablet ef76600ded194f318cf97f157845bea8: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.047253 14056 consensus_queue.cc:1035] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.050745 13922 consensus_queue.cc:1035] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.064800 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.064925 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.066797 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.067298 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d78407a3354848826ab84e5b750ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:22.067605 14078 tablet_replica.cc:330] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:22.067606 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d78407a3354848826ab84e5b750ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:22.067775 14078 raft_consensus.cc:2229] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:22.067885 14079 tablet_replica.cc:330] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:22.067996 14078 raft_consensus.cc:2258] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.068022 14079 raft_consensus.cc:2229] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.068149 14079 raft_consensus.cc:2258] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.068523 14078 ts_tablet_manager.cc:1875] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.068671 14079 ts_tablet_manager.cc:1875] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.070245 14078 ts_tablet_manager.cc:1888] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.070364 14078 log.cc:1196] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.070679 14078 ts_tablet_manager.cc:1909] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:22.070676 14079 ts_tablet_manager.cc:1888] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.070744 14079 log.cc:1196] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.071005 14079 ts_tablet_manager.cc:1909] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:22.071456 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4]) successfully deleted
I20240503 07:59:22.071673 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4]) successfully deleted
I20240503 07:59:22.122449 14080 ts_tablet_manager.cc:902] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:22.122948 14080 tablet_copy_client.cc:308] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:22.123373 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet b61209b73f1f4872b5eccc823b1fa3df from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:38973)
I20240503 07:59:22.123466 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet b61209b73f1f4872b5eccc823b1fa3df from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:38973: session id = a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df
I20240503 07:59:22.124198 6450 tablet_copy_source_session.cc:205] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:22.125017 14080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.126794 14080 tablet_copy_client.cc:783] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:22.126932 14080 tablet_copy_client.cc:647] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:22.128077 14080 tablet_copy_client.cc:523] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:22.129439 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:22.134068 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:22.134274 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:22.134434 14080 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:22.134845 14080 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:22.134964 14080 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:22.135092 14080 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:22.135437 14080 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:22.135808 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df received from {username='slave'} at 127.0.99.132:38973
I20240503 07:59:22.135877 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df on tablet b61209b73f1f4872b5eccc823b1fa3df with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:22.168301 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:22.169857 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.173269 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.173480 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.173539 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.173717 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.175870 14086 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.176039 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.176291 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.176451 14087 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.176851 14086 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.177420 14087 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.178457 14090 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.178481 14086 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.178568 14086 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:22.179028 14086 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.179152 14086 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.179196 14086 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.179339 14086 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.179419 14090 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.179647 14086 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.180301 14087 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.180428 14087 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:22.180757 14090 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.180864 14090 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.180752 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:22.181259 14090 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.181360 14090 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.181385 14090 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.181483 14090 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.181718 14090 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.183933 14087 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.184473 14087 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.184695 14087 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.185016 14087 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.186426 14087 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:22.188561 8957 tablet_service.cc:1467] Processing CreateTablet for tablet d8c681e92f9748f2b6b20958a8efed36 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189078 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c681e92f9748f2b6b20958a8efed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.189323 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6317b68edd7c4512bb96248a9cac9399 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189488 8956 tablet_service.cc:1467] Processing CreateTablet for tablet cd978026a0aa42758f07a701131fdc27 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189545 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6317b68edd7c4512bb96248a9cac9399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.189723 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd978026a0aa42758f07a701131fdc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.191000 14087 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.191434 6539 tablet_service.cc:1467] Processing CreateTablet for tablet d8c681e92f9748f2b6b20958a8efed36 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.191707 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c681e92f9748f2b6b20958a8efed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.192171 8559 tablet_service.cc:1467] Processing CreateTablet for tablet cd978026a0aa42758f07a701131fdc27 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.192358 14087 tablet_bootstrap.cc:654] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.192430 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd978026a0aa42758f07a701131fdc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.193202 14087 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.193253 14087 ts_tablet_manager.cc:1367] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:22.193571 14087 raft_consensus.cc:348] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.193639 14087 raft_consensus.cc:374] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.193657 14087 raft_consensus.cc:729] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.193781 14087 consensus_queue.cc:260] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.193995 14087 ts_tablet_manager.cc:1398] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:22.194133 14087 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.194224 14086 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.194307 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 6317b68edd7c4512bb96248a9cac9399 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.194569 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6317b68edd7c4512bb96248a9cac9399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.194924 14087 tablet_bootstrap.cc:654] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.195135 14086 tablet_bootstrap.cc:654] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.195735 14090 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.196113 14086 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.196167 14086 ts_tablet_manager.cc:1367] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:22.196478 14086 raft_consensus.cc:348] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.196543 14086 raft_consensus.cc:374] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.196569 14086 raft_consensus.cc:729] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.196655 14086 consensus_queue.cc:260] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.196702 14090 tablet_bootstrap.cc:654] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.196921 14096 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:22.196985 14086 ts_tablet_manager.cc:1398] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.197717 14096 tablet_bootstrap.cc:654] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.198264 14090 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.198350 14090 ts_tablet_manager.cc:1367] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:22.198802 14090 raft_consensus.cc:348] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.198906 14090 raft_consensus.cc:374] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.198940 14090 raft_consensus.cc:729] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.199061 14090 consensus_queue.cc:260] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.199151 14096 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:22.199344 14090 ts_tablet_manager.cc:1398] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.199218 14096 ts_tablet_manager.cc:1367] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.200155 14087 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.200235 14087 ts_tablet_manager.cc:1367] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:22.200642 14087 raft_consensus.cc:348] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.200738 14087 raft_consensus.cc:374] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.200795 14087 raft_consensus.cc:729] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.200915 14087 consensus_queue.cc:260] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.201110 14096 raft_consensus.cc:348] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.201176 14087 ts_tablet_manager.cc:1398] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.201217 14096 raft_consensus.cc:374] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.201248 14096 raft_consensus.cc:729] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.201668 14093 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:22.201346 14096 consensus_queue.cc:260] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.203207 14096 ts_tablet_manager.cc:1398] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:22.203512 14093 tablet_bootstrap.cc:654] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.205183 14093 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:22.205296 14093 ts_tablet_manager.cc:1367] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:22.205695 14093 raft_consensus.cc:348] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.205791 14093 raft_consensus.cc:374] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.205828 14093 raft_consensus.cc:729] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.205955 14093 consensus_queue.cc:260] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.206243 14093 ts_tablet_manager.cc:1398] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.219100 12930 raft_consensus.cc:482] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.219172 12930 raft_consensus.cc:504] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.219487 12930 leader_election.cc:288] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.219748 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:22.219767 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:22.219843 8843 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.219869 8581 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.220104 8912 leader_election.cc:302] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:22.220257 12930 raft_consensus.cc:2781] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.220306 12930 raft_consensus.cc:482] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.220340 12930 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.221007 12930 raft_consensus.cc:504] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.221298 12930 leader_election.cc:288] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.221590 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:22.221696 8843 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.221771 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:22.221887 8581 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.222442 8843 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.222733 8912 leader_election.cc:302] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:22.222865 8581 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.222903 12930 raft_consensus.cc:2781] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.222972 12930 raft_consensus.cc:686] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:22.223083 12930 consensus_queue.cc:237] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.223371 13922 raft_consensus.cc:482] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.223453 13922 raft_consensus.cc:504] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.224298 13922 leader_election.cc:288] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:22.224668 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd978026a0aa42758f07a701131fdc27" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:22.224810 8975 raft_consensus.cc:2454] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:22.225113 8516 leader_election.cc:302] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
W20240503 07:59:22.225193 8517 leader_election.cc:334] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.225100 6151 catalog_manager.cc:5484] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:22.225247 13922 raft_consensus.cc:2781] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.225304 13922 raft_consensus.cc:482] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.225355 13922 raft_consensus.cc:3037] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.226104 13922 raft_consensus.cc:504] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.226380 13922 leader_election.cc:288] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:22.226680 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd978026a0aa42758f07a701131fdc27" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:22.226771 8975 raft_consensus.cc:3037] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.227499 8517 leader_election.cc:334] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.227643 8975 raft_consensus.cc:2454] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:22.227908 8516 leader_election.cc:302] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.228294 13922 raft_consensus.cc:2781] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.228385 13922 raft_consensus.cc:686] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:22.228487 13922 consensus_queue.cc:237] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.230325 6151 catalog_manager.cc:5484] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:22.246703 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.259523 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.262571 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:22.312248 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.315135 12930 raft_consensus.cc:482] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.315225 12930 raft_consensus.cc:504] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.315567 12930 leader_election.cc:288] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:22.315860 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c681e92f9748f2b6b20958a8efed36" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:22.315996 6559 raft_consensus.cc:2454] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.316285 8910 leader_election.cc:302] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:22.316459 12930 raft_consensus.cc:2781] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.316519 12930 raft_consensus.cc:482] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.316560 12930 raft_consensus.cc:3037] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.316584 8913 leader_election.cc:334] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.317337 12930 raft_consensus.cc:504] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.317607 12930 leader_election.cc:288] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:22.317870 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c681e92f9748f2b6b20958a8efed36" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:22.317970 6559 raft_consensus.cc:3037] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.318491 8913 leader_election.cc:334] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.318809 6559 raft_consensus.cc:2454] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.319053 8910 leader_election.cc:302] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.319211 12930 raft_consensus.cc:2781] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.319298 12930 raft_consensus.cc:686] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:22.319437 12930 consensus_queue.cc:237] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.321723 6156 catalog_manager.cc:5484] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.334833 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:22.337989 8581 raft_consensus.cc:1261] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.338047 8843 raft_consensus.cc:1261] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.338348 12930 consensus_queue.cc:1035] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.338482 11524 consensus_queue.cc:1035] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.340962 6156 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
I20240503 07:59:22.341604 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
I20240503 07:59:22.341991 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.342268 14062 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:22.343001 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.343252 14065 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:22.344327 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.345428 14107 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.345839 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 1)
I20240503 07:59:22.348187 6151 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
W20240503 07:59:22.361014 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.373528 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.450675 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:22.455861 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.456004 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.456187 14062 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:22.456323 14107 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.457273 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.457569 14065 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
W20240503 07:59:22.459273 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.461278 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 2)
W20240503 07:59:22.491883 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.498047 6690 raft_consensus.cc:1203] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Deduplicated request from leader. Original: 1.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 07:59:22.508476 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.544335 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.562091 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:22.567549 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.568521 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:22.569221 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.572530 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.573702 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 3)
I20240503 07:59:22.594962 14108 raft_consensus.cc:482] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.595068 14108 raft_consensus.cc:504] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.595415 14108 leader_election.cc:288] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:22.595978 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6317b68edd7c4512bb96248a9cac9399" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:22.596104 6298 raft_consensus.cc:2454] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:22.596403 8776 leader_election.cc:302] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:22.596571 14108 raft_consensus.cc:2781] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 07:59:22.596596 8779 leader_election.cc:334] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.596632 14108 raft_consensus.cc:482] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.596688 14108 raft_consensus.cc:3037] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.597697 14108 raft_consensus.cc:504] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.598016 14108 leader_election.cc:288] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:22.598287 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6317b68edd7c4512bb96248a9cac9399" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:22.598397 6298 raft_consensus.cc:3037] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.598940 8779 leader_election.cc:334] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.599237 6298 raft_consensus.cc:2454] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:22.599494 8776 leader_election.cc:302] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.599642 14108 raft_consensus.cc:2781] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.599711 14108 raft_consensus.cc:686] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:22.599864 14108 consensus_queue.cc:237] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.601778 6154 catalog_manager.cc:5484] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:22.627704 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.629621 14070 consensus_queue.cc:1035] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:22.633198 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:22.638283 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.677311 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.677446 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.679280 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.679746 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:22.679832 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:22.680058 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:22.680159 14112 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:22.680248 14113 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:22.680346 14112 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.680361 14114 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:22.680410 14113 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.680477 14114 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:22.680483 14112 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.680536 14113 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.680656 14114 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.681011 14112 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.681188 14113 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.681191 14114 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.683362 14112 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683393 14114 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683393 14113 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683454 14113 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683454 14114 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683477 14112 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683727 14113 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:22.683727 14114 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:22.683751 14112 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:22.684304 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
I20240503 07:59:22.684459 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
I20240503 07:59:22.684469 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
W20240503 07:59:22.731793 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.750429 14106 consensus_queue.cc:1035] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:22.754549 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.781143 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:22.781433 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.786182 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786188 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786392 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786443 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.786458 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.786593 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.788123 14117 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.788870 14119 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:22.788924 14118 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.789103 14117 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.789631 14119 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.789988 14118 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.790270 14117 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.790376 14117 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:22.790653 14119 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:22.790731 14119 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.790866 14117 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.790982 14117 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.791018 14117 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.791132 14117 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791204 14119 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791285 14119 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.791312 14119 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.791407 14117 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.791425 14119 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791663 14119 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.793313 14118 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.793519 14118 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:22.797345 14118 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.797463 14118 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.797508 14118 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.797632 14118 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.797910 14118 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240503 07:59:22.839605 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.867251 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.911473 13018 raft_consensus.cc:482] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.911594 13018 raft_consensus.cc:504] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.911927 13018 leader_election.cc:288] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.912199 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:22.912243 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:22.912362 8843 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:22.912372 8581 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:22.912621 6494 leader_election.cc:302] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:22.912801 13018 raft_consensus.cc:2781] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.912865 13018 raft_consensus.cc:482] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.912914 13018 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.913751 13018 raft_consensus.cc:504] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.914072 13018 leader_election.cc:288] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.914427 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:22.914544 8843 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.914861 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:22.914958 8581 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.915359 8843 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:22.915637 6496 leader_election.cc:302] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:22.915740 8581 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:22.915803 13018 raft_consensus.cc:2781] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.915896 13018 raft_consensus.cc:686] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:22.916023 13018 consensus_queue.cc:237] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.918051 6154 catalog_manager.cc:5484] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.956140 14109 raft_consensus.cc:1050] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:22.956606 14109 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:22.958097 6690 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:22.958139 8843 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:22.958770 12434 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:22.959033 12434 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:22.960264 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.961519 12434 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.963371 8843 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.963627 6690 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.966161 6154 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:22.983368 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 551f25dcee034b7894498e3c748b0f0e: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.006405 12241 consensus_queue.cc:1035] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.009367 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:23.037837 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.041615 8581 raft_consensus.cc:1261] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.041718 8843 raft_consensus.cc:1261] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.042023 13018 consensus_queue.cc:1035] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.042301 13018 consensus_queue.cc:1035] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.045094 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.045352 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.045655 14107 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.046299 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.046572 14065 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:23.048272 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.049297 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.053551 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 1)
I20240503 07:59:23.053885 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.054746 14137 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.147296 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:23.153956 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154016 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154248 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.154284 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.154399 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154628 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157052 14138 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.157387 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:23.157423 8559 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.157557 6277 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.157678 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157774 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157811 14139 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:23.158317 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.158538 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.158701 14139 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.159637 14118 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.159776 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.159976 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.159922 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.160156 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.160514 14118 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.160631 8824 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.160902 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.161864 14118 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.161953 14118 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.162249 14119 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:23.162340 14118 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.162431 14118 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.162458 14118 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.162586 14118 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.162849 14118 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.163009 14118 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.163043 14119 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.163803 14118 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.163956 14117 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:23.164856 14117 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.165864 14138 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.167317 14119 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:23.167385 14119 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:23.167742 14119 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.167826 14119 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.167853 14119 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.167968 14119 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.168195 14142 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.168215 14119 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.169176 14142 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.169652 14138 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.169773 14138 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:23.169800 14140 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.170226 14138 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.170389 14138 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.170521 14138 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.170701 14140 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.170709 14138 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.171747 14138 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.171984 14118 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.172078 14118 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:23.172484 14118 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.172575 14118 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.172603 14118 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.172693 14118 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.172976 14118 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.173900 14139 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:23.173986 14139 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:23.174392 14139 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.174491 14139 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.174522 14139 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.174643 14139 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.174901 14139 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.176604 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.176954 14137 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.177275 14142 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.177349 14142 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:23.177542 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.177785 14107 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.177752 14142 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.177843 14142 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.177874 14142 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.177980 14142 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.178231 14142 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.178411 14142 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.178553 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.178777 14065 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:23.178828 14117 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:23.178891 14117 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240503 07:59:23.179304 14117 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.179404 14117 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.179441 14117 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.179546 14117 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.179782 14117 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.181383 14142 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.182521 14140 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.182636 14140 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:23.183087 14140 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.183231 14140 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.183296 14140 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.183452 14140 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.183789 14140 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.184818 14142 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.184942 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 2)
I20240503 07:59:23.185173 14142 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:23.185603 14142 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.185760 14142 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.185830 14142 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.185978 14142 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.186542 14142 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.191937 14146 raft_consensus.cc:482] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.192009 14146 raft_consensus.cc:504] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.192354 14146 leader_election.cc:288] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:23.192569 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:23.192632 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:23.192682 8843 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.192739 6298 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.192970 8516 leader_election.cc:302] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.193353 14146 raft_consensus.cc:2781] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.193414 14146 raft_consensus.cc:482] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.193459 14146 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.194120 14146 raft_consensus.cc:504] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.194401 14146 leader_election.cc:288] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:23.194675 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:23.194775 6298 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.194814 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:23.194891 8843 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.195627 6298 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.195627 8843 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.195915 8516 leader_election.cc:302] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.196051 14146 raft_consensus.cc:2781] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.196131 14146 raft_consensus.cc:686] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.196242 14146 consensus_queue.cc:237] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.197997 6155 catalog_manager.cc:5484] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:23.205488 14063 raft_consensus.cc:482] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.205569 14063 raft_consensus.cc:504] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.205947 14063 leader_election.cc:288] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:23.206161 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:23.206214 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:23.206280 6559 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.206313 6430 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.206503 6232 leader_election.cc:302] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.206650 14063 raft_consensus.cc:2781] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.206705 14063 raft_consensus.cc:482] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.206749 14063 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.207533 14063 raft_consensus.cc:504] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.207850 14063 leader_election.cc:288] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:23.208093 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:23.208187 6559 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.208250 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:23.208323 6430 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.208809 6559 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.209012 6430 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.209077 6232 leader_election.cc:302] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.209226 14063 raft_consensus.cc:2781] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.209297 14063 raft_consensus.cc:686] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:23.209411 14063 consensus_queue.cc:237] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.210974 6155 catalog_manager.cc:5484] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.255878 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.262044 14146 raft_consensus.cc:482] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.262133 14146 raft_consensus.cc:504] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.262485 14146 leader_election.cc:288] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:23.262737 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.262784 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:23.262866 8976 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.262905 6691 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.263162 8516 leader_election.cc:302] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.263331 14146 raft_consensus.cc:2781] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.263389 14146 raft_consensus.cc:482] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.263432 14146 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.264115 14146 raft_consensus.cc:504] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.264393 14146 leader_election.cc:288] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:23.264657 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.264699 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:23.264747 8976 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.264813 6691 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.265497 8976 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.265610 6691 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.265745 8516 leader_election.cc:302] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.266017 14146 raft_consensus.cc:2781] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.266120 14146 raft_consensus.cc:686] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.266486 14146 consensus_queue.cc:237] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.268529 6155 catalog_manager.cc:5484] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:23.287719 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:23.292295 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.292919 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.296203 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.297508 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 3)
W20240503 07:59:23.359516 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.399442 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.399592 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.401573 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.402029 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:23.402151 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:23.402402 14156 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:23.402452 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:23.402558 14157 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:23.402570 14156 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:23.402696 14157 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.402719 14158 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:23.402772 14156 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.402824 14157 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.402825 14158 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.402940 14158 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.403419 14157 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.403465 14156 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.403525 14158 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.405484 14156 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405560 14156 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405587 14157 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405587 14158 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405658 14158 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405690 14157 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405906 14156 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:23.405942 14158 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:23.405942 14157 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:23.406617 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.406734 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.406872 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.503042 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:23.508054 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.508037 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.508294 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.508299 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.510016 14117 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:23.510066 14138 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.510180 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.510378 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.510833 14117 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.510862 14138 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.512027 14117 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:23.512105 14117 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:23.512151 14138 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.512223 14138 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.512363 14118 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.512518 14117 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512610 14117 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.512640 14117 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.512583 14138 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512676 14138 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.512696 14138 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.512756 14117 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512951 14138 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.513053 14117 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.513228 14118 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.513264 14138 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.514600 14118 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.514676 14118 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:23.515009 14118 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.515093 14118 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.515122 14118 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.515219 14118 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.515460 14118 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.590819 14124 consensus_queue.cc:1035] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.593855 14124 consensus_queue.cc:1035] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.616364 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.625484 14146 consensus_queue.cc:1035] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.629092 14070 consensus_queue.cc:1035] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.654918 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.668956 14146 consensus_queue.cc:1035] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.671737 14170 consensus_queue.cc:1035] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 07:59:23.740381 14170 raft_consensus.cc:482] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.740481 14170 raft_consensus.cc:504] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.740803 14170 leader_election.cc:288] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.741061 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:23.741147 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.741194 8843 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.741284 8975 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.741573 8516 leader_election.cc:302] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.741833 14170 raft_consensus.cc:2781] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.741890 14170 raft_consensus.cc:482] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.741916 14170 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.742870 14170 raft_consensus.cc:504] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.743153 14170 leader_election.cc:288] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.743422 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:23.743530 8843 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.743513 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.743619 8975 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.744266 8843 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.744422 8975 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.744503 8516 leader_election.cc:302] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.744647 14170 raft_consensus.cc:2781] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.744735 14170 raft_consensus.cc:686] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.744881 14170 consensus_queue.cc:237] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.747112 6154 catalog_manager.cc:5484] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.753748 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.779624 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:23.785744 8560 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.785768 8956 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786029 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786051 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786434 8957 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786478 8955 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786636 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786756 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.788132 14118 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.788729 14138 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.789206 14118 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.789577 14138 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.789763 6408 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.789960 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.789909 6410 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.790124 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.790604 14138 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.790669 14138 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.791074 14138 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.791169 14138 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.791203 14138 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.791316 14138 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.791577 14138 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.791761 14138 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.792454 14118 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.792533 14118 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:23.792570 14138 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.792990 14118 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.793085 14118 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.793112 14118 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.793211 14118 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.793212 6539 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.793449 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.793488 14118 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.793489 6279 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.793715 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.795363 14180 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:23.795452 14181 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.796252 14180 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.796960 14138 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.797021 14181 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.797046 14138 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:23.797472 14138 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.797555 14138 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.797582 14138 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.797696 14138 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.797953 14138 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.798076 14178 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.798282 14138 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.798415 8560 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.798619 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.799047 14138 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.799418 14178 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.800001 14138 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.800074 14138 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.800271 14118 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.800866 14138 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.801029 14138 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.801075 14118 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.801097 14138 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.801244 14138 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.801398 14181 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.801472 14181 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:23.801671 14138 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.801877 14181 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.802011 14181 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.802042 14178 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.802070 14181 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.802181 14178 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:23.802250 14180 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:23.802217 14181 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.802572 14181 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.802637 14178 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.802752 14178 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.802798 14178 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803155 14118 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.802382 14180 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:23.803237 14118 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:23.803640 14118 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.803705 14118 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.803723 14118 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803663 14180 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.803755 14180 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.803787 14180 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803808 14118 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.803917 14180 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.804008 14118 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.804191 14180 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.805323 14178 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.805665 14178 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:23.805859 14178 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.806788 14178 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.807799 14178 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.807875 14178 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.808197 14178 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.808310 14178 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.808348 14178 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.808454 14178 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.808707 14178 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.855181 14179 raft_consensus.cc:482] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.855266 14179 raft_consensus.cc:504] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.855651 14179 leader_election.cc:288] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.855952 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:23.855957 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.856067 8581 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.856067 8975 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.856323 6363 leader_election.cc:302] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.856482 14179 raft_consensus.cc:2781] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.856535 14179 raft_consensus.cc:482] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.856573 14179 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.857582 14179 raft_consensus.cc:504] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.857918 14179 leader_election.cc:288] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.858136 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:23.858233 8581 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.858237 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.858351 8975 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.858974 8581 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.859032 8975 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.859205 6363 leader_election.cc:302] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.859397 14179 raft_consensus.cc:2781] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.859475 14179 raft_consensus.cc:686] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:23.859611 14179 consensus_queue.cc:237] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.861539 6154 catalog_manager.cc:5484] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:23.864531 14179 raft_consensus.cc:482] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.864609 14179 raft_consensus.cc:504] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.864946 14179 leader_election.cc:288] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:23.865188 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.865283 8975 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.865260 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:23.865418 6559 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.865499 6365 leader_election.cc:302] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.865645 14179 raft_consensus.cc:2781] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.865700 14179 raft_consensus.cc:482] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.865739 14179 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.866480 14179 raft_consensus.cc:504] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.866848 14179 leader_election.cc:288] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:23.867085 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.867127 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:23.867194 8975 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.867231 6559 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.867892 8975 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.868028 6559 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.868202 6365 leader_election.cc:302] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.868359 14179 raft_consensus.cc:2781] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.868448 14179 raft_consensus.cc:686] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:23.868570 14179 consensus_queue.cc:237] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.870329 6154 catalog_manager.cc:5484] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:23.893488 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.900075 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.903124 8975 raft_consensus.cc:1261] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.903199 8843 raft_consensus.cc:1261] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.903575 14146 consensus_queue.cc:1035] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.903717 14170 consensus_queue.cc:1035] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.905850 14008 mvcc.cc:205] Tried to move back new op lower bound from 7023506079344590848 to 7023506078700249088. Current Snapshot: MvccSnapshot[applied={T|T < 7023506079344590848}]
I20240503 07:59:23.906531 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.907151 14194 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.907435 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.907737 6155 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.907872 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.908061 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.908200 14195 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:23.910342 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.912112 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 1)
I20240503 07:59:23.912353 14193 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
W20240503 07:59:23.937774 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.941078 14124 raft_consensus.cc:482] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.941161 14124 raft_consensus.cc:504] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.941468 14124 leader_election.cc:288] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.941766 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.941766 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:23.941896 8975 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.941896 8581 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.942194 6232 leader_election.cc:302] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.942395 14124 raft_consensus.cc:2781] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.942462 14124 raft_consensus.cc:482] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.942507 14124 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.943408 14124 raft_consensus.cc:504] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.943783 14124 leader_election.cc:288] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.944092 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:23.944095 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.944191 8975 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.944195 8581 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.945104 8975 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.945120 8581 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.945403 6234 leader_election.cc:302] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.945590 14124 raft_consensus.cc:2781] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.945700 14124 raft_consensus.cc:686] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:23.945850 14124 consensus_queue.cc:237] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.947878 6155 catalog_manager.cc:5484] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.976306 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:23.997613 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.015707 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.020413 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.020648 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.020718 14194 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.020833 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.021157 14193 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:24.021179 14195 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:24.025437 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 2)
I20240503 07:59:24.128479 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:24.133468 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.133967 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.135344 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.140028 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 3)
W20240503 07:59:24.144037 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:24.152984 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:24.174988 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.243746 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.244016 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.246352 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.246896 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:24.246925 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:24.247172 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:24.247349 14200 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:24.247383 14199 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:24.247464 14201 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:24.247483 14200 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.247537 14199 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.247609 14201 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:24.247646 14199 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.247642 14200 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.247840 14201 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.248234 14199 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.248577 14200 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.249198 14201 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.250945 14199 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.251055 14199 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.251325 14199 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:24.251897 14201 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.251976 14201 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.252249 14201 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:24.253146 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
I20240503 07:59:24.255262 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
W20240503 07:59:24.256101 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.256932 14200 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.257050 14200 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.257628 14200 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:24.258281 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
I20240503 07:59:24.274338 14179 consensus_queue.cc:1035] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.278115 14204 consensus_queue.cc:1035] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:24.284508 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:24.339026 14063 consensus_queue.cc:1035] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.341678 14063 consensus_queue.cc:1035] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.347896 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:24.352699 6539 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.352983 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.352964 8824 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.353169 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:24.355262 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:24.355491 14212 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.355468 6279 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.355676 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.356072 14211 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:24.356462 14212 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.357008 14211 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.358395 14212 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.358544 14212 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.358927 14215 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:24.358959 14212 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.359097 14212 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.359155 14212 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.359289 14212 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.359472 14211 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:24.359567 14212 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.359584 14211 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.359988 14211 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.360126 14211 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.360191 14211 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.360414 14215 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.360357 14211 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.361385 14211 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.361927 14215 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:24.362124 14215 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.362504 14215 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.362655 14215 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.362727 14215 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.362887 14215 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.363322 14215 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:24.386463 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.398824 14204 consensus_queue.cc:1035] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.401899 14109 consensus_queue.cc:1035] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:24.477234 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.478379 14168 raft_consensus.cc:482] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.478463 14168 raft_consensus.cc:504] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.478781 14168 leader_election.cc:288] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:24.479139 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:24.479183 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:24.479292 6560 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.479298 6296 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.479537 8776 leader_election.cc:302] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:24.479713 14168 raft_consensus.cc:2781] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.479782 14168 raft_consensus.cc:482] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.479822 14168 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.480655 14168 raft_consensus.cc:504] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.480994 14168 leader_election.cc:288] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:24.481323 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:24.481367 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:24.481446 6560 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.481458 6296 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.481788 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:24.482415 6560 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.482414 6296 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.482754 8776 leader_election.cc:302] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:24.482921 14168 raft_consensus.cc:2781] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.483032 14168 raft_consensus.cc:686] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:24.483179 14168 consensus_queue.cc:237] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.487632 6155 catalog_manager.cc:5484] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.488113 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.488256 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.488371 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.488529 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.490689 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490810 14222 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:24.490808 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490808 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490926 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491046 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491047 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491091 14221 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.493046 14212 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.493067 14222 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.493319 14221 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.493328 8822 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.493556 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.493790 14224 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:24.493968 14212 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.494659 14224 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.494776 14221 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.494859 14221 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.495280 14221 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.495369 14221 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.495397 14221 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.495406 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.495486 14221 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.495618 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.495638 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.495739 14221 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.495987 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.497591 14221 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.498025 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.498262 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.498415 14221 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.498629 14212 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.498682 14212 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:24.499032 14212 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.499135 14212 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.499174 14212 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.499287 14212 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.499434 14221 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.499519 14221 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.499540 14212 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.499677 14212 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.499907 14221 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.500011 14221 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.500052 14221 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.500167 14221 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.500324 14211 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:24.500380 14212 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.500414 14221 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.500551 14221 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.501209 14211 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.501304 14223 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:24.501353 14221 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.502457 14223 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.502758 14212 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.502835 14212 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.503216 14212 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.503297 14212 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.503337 14212 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.503435 14212 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.503666 14222 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:24.503666 14212 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.503737 14222 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:24.504016 14221 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.504124 14222 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.504228 14222 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.504238 14221 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.504264 14222 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.504370 14222 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.504611 14222 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.504627 14221 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.504722 14221 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.504752 14221 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.505400 14221 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.505733 14221 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.506701 14211 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:24.506780 14211 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:24.507184 14211 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.507285 14211 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.507314 14211 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.507412 14211 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.507651 14211 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:24.507768 14224 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:24.507848 14224 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:24.507978 14223 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:24.508049 14223 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:24.508266 14224 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.508591 14224 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.508625 14224 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.508863 14224 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.509140 14224 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.508452 14223 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.511906 14223 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.511957 14223 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.512066 14223 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.512292 14223 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:24.536136 14168 raft_consensus.cc:482] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.536228 14168 raft_consensus.cc:504] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.536584 14168 leader_election.cc:288] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:24.536862 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:24.536917 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:24.536995 6690 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.537025 6429 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.537285 8779 leader_election.cc:302] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:24.537474 14174 raft_consensus.cc:2781] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.537534 14174 raft_consensus.cc:482] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.537572 14174 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.538463 14174 raft_consensus.cc:504] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.538789 14174 leader_election.cc:288] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:24.539076 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:24.539120 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:24.539178 6690 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.539201 6429 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.539958 6429 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.540015 6690 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.540277 8778 leader_election.cc:302] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.540488 14174 raft_consensus.cc:2781] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.540563 14174 raft_consensus.cc:686] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:24.540705 14174 consensus_queue.cc:237] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.543228 6155 catalog_manager.cc:5484] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:24.546839 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.575613 14197 raft_consensus.cc:482] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.575724 14197 raft_consensus.cc:504] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.576092 14197 leader_election.cc:288] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:24.576323 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:24.576450 8581 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:24.576463 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:24.576567 8842 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:24.576737 6625 leader_election.cc:302] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:24.576931 14197 raft_consensus.cc:2781] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.576984 14197 raft_consensus.cc:482] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.577041 14106 raft_consensus.cc:482] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.577103 14197 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.577121 14106 raft_consensus.cc:504] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.577389 14106 leader_election.cc:288] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:24.577697 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:24.577757 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:24.577826 6690 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:24.577883 6560 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:24.578117 8912 leader_election.cc:302] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.578146 14197 raft_consensus.cc:504] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.578451 14197 leader_election.cc:288] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:24.578485 14106 raft_consensus.cc:2781] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.578536 14106 raft_consensus.cc:482] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.578570 14106 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.578687 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:24.578773 8581 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.579464 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:24.579576 8842 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.579555 14106 raft_consensus.cc:504] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.579607 8581 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:24.579869 14106 leader_election.cc:288] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:24.579849 6625 leader_election.cc:302] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:24.580004 14197 raft_consensus.cc:2781] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.580199 14197 raft_consensus.cc:686] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:24.580305 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:24.580336 14197 consensus_queue.cc:237] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.580408 6690 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.580394 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:24.580489 6560 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.580524 8842 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:24.581488 6690 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:24.581502 6560 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:24.581743 8912 leader_election.cc:302] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.582321 14106 raft_consensus.cc:2781] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.582515 14106 raft_consensus.cc:686] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:24.582648 14106 consensus_queue.cc:237] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.583223 6155 catalog_manager.cc:5484] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.584947 6156 catalog_manager.cc:5484] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:24.605856 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:24.609153 6560 raft_consensus.cc:1261] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.609153 6296 raft_consensus.cc:1261] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.609511 14174 consensus_queue.cc:1035] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.609660 14168 consensus_queue.cc:1035] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.612165 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.612334 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]). Expected version 1 got 0
I20240503 07:59:24.612458 14194 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.613233 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]). Expected version 1 got 0
I20240503 07:59:24.616619 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 1)
I20240503 07:59:24.616900 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.617666 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.617437 14240 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.620061 14022 mvcc.cc:205] Tried to move back new op lower bound from 7023506082235392000 to 7023506081724542976. Current Snapshot: MvccSnapshot[applied={T|T < 7023506082235392000}]
I20240503 07:59:24.617954 14241 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
W20240503 07:59:24.701854 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.718534 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.723470 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.723784 14194 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.724006 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.724435 14241 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:24.724607 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.724923 14240 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.729859 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 2)
W20240503 07:59:24.749282 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:24.788164 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.808043 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.832820 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 07:59:24.834463 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.837617 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.837913 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.838910 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:24.843109 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.844004 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 3)
W20240503 07:59:24.875711 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.897498 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.939028 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240503 07:59:24.946171 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.946300 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.948482 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.948964 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:24.948964 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:24.949074 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:24.949378 14249 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:24.949419 14247 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:24.949497 14249 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.949573 14247 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:24.949613 14249 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.949750 14247 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.949977 14248 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:24.950119 14248 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.950241 14248 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.950263 14249 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.950264 14247 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.950850 14248 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.952242 14249 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.952323 14249 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.952477 14247 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.952553 14247 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.952664 14249 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:24.952862 14247 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:24.953094 14248 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.953171 14248 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.953356 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
I20240503 07:59:24.953493 14248 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:24.953665 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
I20240503 07:59:24.954052 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
W20240503 07:59:24.987248 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 07:59:24.993680 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 07:59:25.034438 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 07:59:25.044414 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.046444 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240503 07:59:25.050552 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:25.051028 14197 consensus_queue.cc:1035] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.054541 14236 consensus_queue.cc:1035] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.056849 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.056833 14197 consensus_queue.cc:1035] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.057080 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.056741 6539 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.058595 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.059300 14256 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.059496 8560 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.059705 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.060494 14256 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.062611 14259 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.063511 14259 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.065279 14257 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.066358 14257 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.066689 14236 consensus_queue.cc:1035] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.067581 14257 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:25.067706 14257 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:25.067986 14259 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.068125 14259 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:25.068176 14257 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.068274 14257 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.068307 14257 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.068434 14257 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.068694 14257 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.068881 14259 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.069031 14259 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.069108 14259 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.069263 14259 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.069582 14259 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.073278 14256 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.073401 14256 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:25.073989 14256 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.074106 14256 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.074146 14256 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.074255 14256 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.074507 14256 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.085692 14168 consensus_queue.cc:1035] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.088450 14174 consensus_queue.cc:1035] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.101797 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:25.106851 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 07:59:25.107749 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.107720 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.107998 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.108079 6408 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.108083 8957 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.108279 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.108330 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.111019 14268 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:25.110997 8822 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.111245 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.111835 14268 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.112921 14268 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:25.112980 14268 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.113399 14256 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.113399 14268 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.113493 14268 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.113524 14268 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.113636 14268 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.113986 14268 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:25.114035 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 07:59:25.114421 14256 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.114732 6279 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.114812 8560 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.114944 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.115088 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.116456 14256 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.116551 14256 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.116849 14272 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.117031 14256 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.117136 14256 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.117167 14256 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.117286 14259 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.117277 14256 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.117857 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.118067 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.118237 6277 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.118321 14259 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.118419 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.118674 14256 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.119079 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.119288 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.119400 14259 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.119470 14259 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:25.119902 14259 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.120013 14259 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.120047 14259 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.120167 14259 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.120458 14259 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.120827 14269 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.121495 14272 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.123064 14259 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.123142 14269 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.124222 14259 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:25.124986 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.125480 14259 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.125566 14259 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:25.125936 14269 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.125921 14259 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.125991 14269 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:25.125988 14259 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.126013 14259 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.126112 14259 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.126318 14269 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.126368 14259 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.126392 14269 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.126412 14269 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.126518 14269 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.126856 14269 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.127167 14272 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.127242 14272 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:25.127643 14272 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.127734 14272 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.127764 14272 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.127867 14269 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.127866 14272 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.128120 14272 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.128262 14272 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.128805 14269 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.129067 14272 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.132123 14269 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.132238 14269 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:25.132990 14269 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.133133 14269 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.133208 14269 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.133447 14269 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.134438 14269 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.135030 14272 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.135118 14272 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:25.135473 14272 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.135560 14272 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.135591 14272 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.135710 14272 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.135985 14272 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.136183 14272 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.137082 14272 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:25.138739 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.139338 14272 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.139417 14272 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.139827 14272 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.139910 14272 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.139942 14272 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.140060 14272 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.140283 14272 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:25.148039 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:25.162971 14243 raft_consensus.cc:482] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.163064 14243 raft_consensus.cc:504] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.163400 14243 leader_election.cc:288] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.163664 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.163704 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.163790 8843 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:25.163805 8581 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:25.164098 6496 leader_election.cc:302] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:25.164461 14243 raft_consensus.cc:2781] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.164528 14243 raft_consensus.cc:482] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.164570 14243 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.165637 14243 raft_consensus.cc:504] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.165971 14243 leader_election.cc:288] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.166486 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.166589 8843 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.166594 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.166678 8581 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.167479 8581 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:25.167479 8843 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:25.167821 6494 leader_election.cc:302] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.168282 14243 raft_consensus.cc:2781] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.168363 14243 raft_consensus.cc:686] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:25.168485 14243 consensus_queue.cc:237] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.170495 6156 catalog_manager.cc:5484] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.178460 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 07:59:25.179116 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:25.195617 14245 raft_consensus.cc:482] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.195703 14245 raft_consensus.cc:504] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.196017 14245 leader_election.cc:288] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.196288 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.196328 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:25.196394 8581 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.196471 8976 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.196628 6232 leader_election.cc:302] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.196928 14245 raft_consensus.cc:2781] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.196979 14245 raft_consensus.cc:482] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.197005 14245 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.197770 14245 raft_consensus.cc:504] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.198050 14245 leader_election.cc:288] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.198340 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:25.198355 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.198439 8581 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.198441 8976 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.199187 8581 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.199218 8976 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.199424 6232 leader_election.cc:302] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.199560 14245 raft_consensus.cc:2781] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.199646 14245 raft_consensus.cc:686] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:25.199765 14245 consensus_queue.cc:237] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.201882 6154 catalog_manager.cc:5484] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.239369 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.268378 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.279824 14245 raft_consensus.cc:482] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.279918 14245 raft_consensus.cc:504] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.280264 14245 leader_election.cc:288] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:25.280519 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.280505 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:25.280625 8843 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.280627 6430 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.280886 6235 leader_election.cc:302] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.281077 14245 raft_consensus.cc:2781] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.281137 14245 raft_consensus.cc:482] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.281163 14245 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.281898 14245 raft_consensus.cc:504] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.282166 14245 leader_election.cc:288] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:25.282438 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.282431 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:25.282555 8843 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.282538 6430 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.283413 6430 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.283413 8843 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.283674 6234 leader_election.cc:302] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:25.283833 14245 raft_consensus.cc:2781] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.283951 14245 raft_consensus.cc:686] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:25.284085 14245 consensus_queue.cc:237] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.286360 6154 catalog_manager.cc:5484] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.293584 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:25.296885 8843 raft_consensus.cc:1261] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.296911 8581 raft_consensus.cc:1261] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.297214 13931 consensus_queue.cc:1035] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.297405 14243 consensus_queue.cc:1035] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.300230 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]). Expected version 1 got 0
I20240503 07:59:25.300283 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.300594 14240 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.301252 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.301618 14286 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:25.302466 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]). Expected version 1 got 0
I20240503 07:59:25.304359 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.304622 14194 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:25.304998 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43] alter complete (version 1)
W20240503 07:59:25.343437 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:25.409979 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:25.415477 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.415791 14240 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.416134 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.416235 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.416461 14194 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:25.416687 14286 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:25.421741 14179 raft_consensus.cc:482] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.421888 14179 raft_consensus.cc:504] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.422243 14179 leader_election.cc:288] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.422667 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:25.422828 6298 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:25.422955 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.423067 8581 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:25.423274 6363 leader_election.cc:302] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.423595 14179 raft_consensus.cc:2781] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.423709 14179 raft_consensus.cc:482] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.423807 14179 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.425302 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43] alter complete (version 2)
I20240503 07:59:25.425459 14179 raft_consensus.cc:504] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.426477 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:25.426594 6298 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.426716 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.426813 8581 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.427562 6298 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:25.427707 8581 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:25.428066 6363 leader_election.cc:302] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.428303 14204 raft_consensus.cc:2781] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.428471 14204 raft_consensus.cc:686] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:25.428615 14204 consensus_queue.cc:237] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.428926 14179 leader_election.cc:288] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.433759 6156 catalog_manager.cc:5484] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.466337 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:25.483806 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.528755 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:25.533466 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.534278 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.536854 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.538676 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43] alter complete (version 3)
W20240503 07:59:25.552088 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.593273 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:25.615561 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:25.627889 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): CreateTablet RPC failed for tablet ef76600ded194f318cf97f157845bea8: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:25.640537 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.640667 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.642556 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.643023 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:25.643035 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:25.643374 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:25.643404 14295 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:25.643417 14296 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:25.643530 14296 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:25.643530 14295 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.643615 14297 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:25.643640 14295 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.643786 14297 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.643847 14296 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.643946 14297 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.644424 14295 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.644579 14296 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.644593 14297 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.646550 14297 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646657 14297 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646770 14296 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646770 14295 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646842 14295 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646888 14296 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646924 14297 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:25.647099 14295 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:25.647166 14296 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:25.647541 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.647701 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.647722 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.723004 14244 consensus_queue.cc:1035] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.726688 14245 consensus_queue.cc:1035] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.744082 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:25.748764 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.748916 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.749027 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.749168 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.751008 14302 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.751251 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.751302 14303 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:25.751489 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.752086 14302 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.752113 14303 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.752947 14302 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.753006 14302 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.753324 14302 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.753397 14302 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.753420 14302 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.753525 14302 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.753571 14306 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.753787 14302 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.754317 14303 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:25.754473 14303 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.754561 14306 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.754849 14303 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.754935 14303 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.754962 14303 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.755087 14303 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.755326 14303 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.756201 14306 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:25.756285 14306 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.756701 14306 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.756886 14306 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.756939 14306 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.757061 14306 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.757401 14306 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.797529 14244 consensus_queue.cc:1035] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.800944 14244 consensus_queue.cc:1035] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.885898 14289 raft_consensus.cc:482] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.886020 14289 raft_consensus.cc:504] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.886505 14289 leader_election.cc:288] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:25.886763 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.886888 8843 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:25.887008 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:25.887121 6627 leader_election.cc:302] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:25.887178 6559 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:25.887259 14289 raft_consensus.cc:2781] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.887308 14289 raft_consensus.cc:482] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.887341 14289 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.888150 14289 raft_consensus.cc:504] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.888451 14289 leader_election.cc:288] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:25.888695 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.888854 8843 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.888875 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:25.888998 6559 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.889868 8843 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:25.889948 6559 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:25.890156 6627 leader_election.cc:302] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:25.890406 14289 raft_consensus.cc:2781] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.890480 14289 raft_consensus.cc:686] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:25.890595 14289 consensus_queue.cc:237] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.892750 6156 catalog_manager.cc:5484] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.908703 14204 consensus_queue.cc:1035] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.911408 14204 consensus_queue.cc:1035] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.983474 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:25.990603 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.990796 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.990942 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.991070 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.991276 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.991544 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.993647 14315 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.993686 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.993880 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.994891 14315 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.995529 14316 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.995744 14306 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.996371 14316 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.996556 14306 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.996995 14315 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.997067 14315 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.997723 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997824 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997881 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997992 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.997982 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.998071 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.998220 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.998268 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.997519 14315 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.000460 14303 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.000469 14302 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.000805 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.001016 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.001339 14302 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.001386 14303 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.001392 14306 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.001458 14306 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:26.001873 14306 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.001991 14306 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.002024 14306 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.002130 14306 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.002214 14315 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.002421 14306 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.002429 14315 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.002557 14315 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.003275 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:26.003414 14315 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:26.004423 14302 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.004514 14302 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:26.005034 14302 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.005133 14302 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.005154 14302 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.005249 14302 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.005504 14302 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.005661 14302 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.006455 14302 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.007603 14316 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.007680 14316 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:26.009286 14316 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.009385 14316 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.009416 14316 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.009528 14316 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.009792 14316 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.009932 14316 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:26.011126 14316 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.013111 14302 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.013190 14302 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:26.013566 8843 raft_consensus.cc:1261] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.013655 6559 raft_consensus.cc:1261] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.013689 14302 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.013772 14302 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.013797 14302 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.013906 14302 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.013911 14197 consensus_queue.cc:1035] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.014058 14251 consensus_queue.cc:1035] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.014146 14302 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.017323 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.017594 14194 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.019572 6156 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]). Expected version 1 got 0
I20240503 07:59:26.020120 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.020416 14328 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.021679 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]). Expected version 1 got 0
I20240503 07:59:26.023641 14303 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.023676 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd] alter complete (version 1)
I20240503 07:59:26.023726 14303 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:26.023861 14319 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:26.024138 14303 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.024250 14303 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.024289 14303 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.024399 14303 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.024681 14303 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.029017 14319 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.030264 14319 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:26.030361 14319 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:26.030475 14316 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.030558 14316 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:26.030807 14319 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.030928 14319 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.030972 14319 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.030941 14316 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031041 14316 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.031075 14316 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.031183 14316 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031188 14319 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031430 14316 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.031514 14319 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.031664 14319 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:26.032470 14319 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.033273 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.033545 14240 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.053018 14319 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:26.053117 14319 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:26.053476 14319 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.053567 14319 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.053596 14319 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.053714 14319 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.053951 14319 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.070333 14291 raft_consensus.cc:482] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.070433 14291 raft_consensus.cc:504] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.070878 14291 leader_election.cc:288] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.071169 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:26.071224 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.071282 8975 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.071352 8581 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.071682 8778 leader_election.cc:302] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:26.071934 14291 raft_consensus.cc:2781] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.072003 14291 raft_consensus.cc:482] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.072036 14291 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.073009 14291 raft_consensus.cc:504] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.073341 14291 leader_election.cc:288] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.073690 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:26.073691 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.073799 8581 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.073799 8975 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.074684 8581 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.074684 8975 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.074942 8776 leader_election.cc:302] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.075091 14291 raft_consensus.cc:2781] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.075232 14291 raft_consensus.cc:686] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.075361 14291 consensus_queue.cc:237] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.077214 6155 catalog_manager.cc:5484] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.125918 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:26.130899 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.131218 14328 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.131282 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.131556 14194 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.133836 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.134127 14240 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.136188 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd] alter complete (version 2)
I20240503 07:59:26.147596 14243 raft_consensus.cc:482] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.147689 14243 raft_consensus.cc:504] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.148269 14243 leader_election.cc:288] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.148471 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:26.148619 6430 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.148975 6497 leader_election.cc:302] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.148975 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.149092 8581 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.149247 13931 raft_consensus.cc:2781] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.149302 13931 raft_consensus.cc:482] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.149355 13931 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.150369 13931 raft_consensus.cc:504] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.150730 13931 leader_election.cc:288] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.151057 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.151168 8581 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.151175 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:26.151296 6430 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.152088 8581 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.152204 6430 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.152324 6494 leader_election.cc:302] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.152580 13931 raft_consensus.cc:2781] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.152668 13931 raft_consensus.cc:686] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:26.152865 13931 consensus_queue.cc:237] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.154770 6154 catalog_manager.cc:5484] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:26.163216 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.163662 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:26.173126 14291 raft_consensus.cc:482] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.173221 14291 raft_consensus.cc:504] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.173542 14291 leader_election.cc:288] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:26.173801 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:26.173858 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:26.173923 6691 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.173970 8975 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.174175 8778 leader_election.cc:302] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:26.174332 14291 raft_consensus.cc:2781] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.174391 14291 raft_consensus.cc:482] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.174435 14291 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.175112 14291 raft_consensus.cc:504] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.175381 14291 leader_election.cc:288] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:26.175652 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:26.175684 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:26.175766 6691 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.175769 8975 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.176432 6691 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.176683 8778 leader_election.cc:302] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:26.176748 8975 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.176966 14291 raft_consensus.cc:2781] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.177055 14291 raft_consensus.cc:686] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.177258 14291 consensus_queue.cc:237] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.179360 6154 catalog_manager.cc:5484] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.239907 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:26.244695 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.245110 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.245254 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.250672 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd] alter complete (version 3)
W20240503 07:59:26.289852 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.354197 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.354370 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.356508 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.357048 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:26.357122 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:26.357144 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:26.357352 14336 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:26.357439 14337 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:26.357503 14336 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.357517 14338 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:26.357574 14337 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:26.357625 14336 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.357636 14338 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.357736 14338 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.357764 14337 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.358294 14337 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.358295 14338 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.358435 14336 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.360155 14336 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360220 14336 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360389 14338 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360389 14337 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360430 14336 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:26.360464 14338 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360464 14337 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360756 14337 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:26.360821 14338 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:26.361248 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.361407 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.361567 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.458215 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:26.463099 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.463950 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.464254 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.464386 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.464632 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.465162 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.465399 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.466892 14303 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.467109 14306 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:26.467768 14303 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.467842 14306 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.468571 14302 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.469465 14302 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.469528 14306 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.469640 14306 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.470198 14303 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.470124 14306 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470249 14306 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.470263 14303 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:26.470286 14306 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.470422 14306 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470635 14303 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470721 14303 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.470741 14303 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.470754 14306 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.470839 14303 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.471078 14303 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.472214 14302 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.472297 14302 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:26.472730 14302 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.473389 14302 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.473431 14302 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.473533 14302 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.473814 14302 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:26.518272 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.550725 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.567178 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:26.568089 14242 consensus_queue.cc:1035] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.568253 14291 consensus_queue.cc:1035] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:26.597110 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.628145 14242 raft_consensus.cc:482] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.628262 14242 raft_consensus.cc:504] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.628643 14242 leader_election.cc:288] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:26.628903 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:26.629067 6559 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.629117 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:26.629240 6690 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.629397 8776 leader_election.cc:302] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.629592 14242 raft_consensus.cc:2781] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.629637 14242 raft_consensus.cc:482] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.629662 14242 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.630439 14242 raft_consensus.cc:504] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.630702 14242 leader_election.cc:288] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:26.630970 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:26.631062 6559 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.631009 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:26.631146 6690 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.631803 6559 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.631938 6690 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.632055 8776 leader_election.cc:302] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.632200 14242 raft_consensus.cc:2781] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.632308 14242 raft_consensus.cc:686] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.632427 14242 consensus_queue.cc:237] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.634423 6156 catalog_manager.cc:5484] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.639158 14242 consensus_queue.cc:1035] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.642360 14351 consensus_queue.cc:1035] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.652436 14243 consensus_queue.cc:1035] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.655193 13931 consensus_queue.cc:1035] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:26.672497 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.698587 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:26.703892 8559 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704046 8824 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704167 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.704154 6670 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704335 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.704383 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.706094 14302 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.706440 14303 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.706410 6408 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.706672 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.706997 14302 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.707257 14360 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:26.707294 14303 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.707984 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.708233 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.708464 14360 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.708909 14363 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:26.709192 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.709466 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.709818 14363 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.709897 14302 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.709968 14302 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:26.710377 14302 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.710482 14302 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.710522 14302 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.710620 14302 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.707984 6279 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.710875 14302 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.710961 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.711449 14306 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:26.711766 14364 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:26.712198 14306 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.712600 14364 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.713320 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.713371 8822 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.713559 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.714365 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.714815 14306 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.714890 14306 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.715286 14306 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.715379 14306 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.715409 14306 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.715515 14306 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.715773 14306 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.716949 14303 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.717031 14303 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:26.717460 14303 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.717558 14303 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.717587 14303 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.717695 14303 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.717788 14363 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:26.717862 14363 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:26.717940 14303 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.718688 14302 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.718935 14363 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.719026 14363 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.719056 14363 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.719188 14363 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.719434 14363 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:26.719514 14302 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.720161 14360 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.720244 14360 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:26.720690 14360 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.720837 14360 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.720901 14360 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.721046 14360 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.721345 14360 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.724042 14302 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.724175 14302 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:26.724342 14364 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:26.724460 14364 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:26.724823 14302 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.724941 14302 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.724992 14302 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.725144 14302 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
W20240503 07:59:26.726137 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.726253 14364 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.726467 14302 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.726478 14364 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.726522 14364 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.726675 14302 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.726662 14364 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.727025 14364 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.727280 14364 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:26.728104 14364 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.728156 14302 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.729203 14364 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:26.729285 14364 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.729683 14364 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.729801 14364 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.729864 14364 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.729992 14364 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.730091 14302 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.730170 14302 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:26.730230 14364 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.730660 14302 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.731333 14302 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.731410 14302 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.731524 14302 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.731783 14302 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:26.744493 14243 raft_consensus.cc:482] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.744602 14243 raft_consensus.cc:504] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.745110 14243 leader_election.cc:288] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.745421 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.745404 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:26.745544 8843 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.745540 6298 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.745796 6496 leader_election.cc:302] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.745963 14243 raft_consensus.cc:2781] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.746019 14243 raft_consensus.cc:482] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.746064 14243 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.746843 14243 raft_consensus.cc:504] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.747190 14243 leader_election.cc:288] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.747469 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.747470 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:26.747567 6298 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.747568 8843 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.748417 6298 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.748452 8843 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.748701 6496 leader_election.cc:302] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.748857 14243 raft_consensus.cc:2781] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.748944 14243 raft_consensus.cc:686] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:26.749063 14243 consensus_queue.cc:237] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.751052 6154 catalog_manager.cc:5484] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.753504 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 07:59:26.756299 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.756747 6560 raft_consensus.cc:1261] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.756822 6690 raft_consensus.cc:1261] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.757153 14242 consensus_queue.cc:1035] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.757297 14351 consensus_queue.cc:1035] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.760497 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.761656 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.761679 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.761713 14218 mvcc.cc:205] Tried to move back new op lower bound from 7023506091031883776 to 7023506090527420416. Current Snapshot: MvccSnapshot[applied={T|T < 7023506091031883776}]
I20240503 07:59:26.762799 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.763051 14377 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.763751 6156 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.763849 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.764164 14376 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.765264 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.766072 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 1)
I20240503 07:59:26.764164 14375 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
W20240503 07:59:26.777298 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.787761 14289 raft_consensus.cc:482] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.787843 14289 raft_consensus.cc:504] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.788200 14289 leader_election.cc:288] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.788527 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.788532 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.788640 8581 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:26.788630 8843 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:26.788924 6625 leader_election.cc:302] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.789250 14197 raft_consensus.cc:2781] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.789314 14197 raft_consensus.cc:482] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.789391 14197 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.790138 14197 raft_consensus.cc:504] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.790413 14197 leader_election.cc:288] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.790673 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.790668 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.790782 8843 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.790781 8581 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.791468 8843 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:26.791491 8581 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:26.791761 6625 leader_election.cc:302] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.791908 14289 raft_consensus.cc:2781] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.791970 14289 raft_consensus.cc:686] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:26.792083 14289 consensus_queue.cc:237] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.794021 6148 catalog_manager.cc:5484] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:26.826154 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:26.842139 14287 raft_consensus.cc:482] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.842235 14287 raft_consensus.cc:504] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.842666 14287 leader_election.cc:288] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:26.842932 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.843005 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:26.843057 8842 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:26.843123 6298 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:26.843327 6365 leader_election.cc:302] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.843600 14287 raft_consensus.cc:2781] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.843662 14287 raft_consensus.cc:482] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.843699 14287 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.844504 14287 raft_consensus.cc:504] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.844825 14287 leader_election.cc:288] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:26.845094 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.845130 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:26.845197 8842 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.845259 6298 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.846105 6298 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:26.846112 8842 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:26.846400 6365 leader_election.cc:302] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.846551 14287 raft_consensus.cc:2781] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.846632 14287 raft_consensus.cc:686] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:26.846820 14287 consensus_queue.cc:237] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.849140 6148 catalog_manager.cc:5484] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:26.869614 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:26.874861 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875108 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875200 14375 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.875391 14377 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.875627 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875926 14376 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.880432 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 2)
W20240503 07:59:26.960742 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:26.962862 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:26.983690 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:26.991406 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.992277 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.992472 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.998104 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 3)
I20240503 07:59:27.102984 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.103133 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.105767 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.106387 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:27.106422 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:27.106671 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:27.106781 14381 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:27.106798 14380 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:27.106932 14382 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:27.106933 14381 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.106932 14380 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.107048 14381 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107048 14380 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107060 14382 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:27.107250 14382 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107785 14382 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.107905 14380 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.107928 14381 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.109881 14382 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.109947 14382 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110059 14381 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.110121 14380 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.110172 14381 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110199 14380 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110257 14382 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:27.110505 14381 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:27.110504 14380 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:27.110849 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
I20240503 07:59:27.111040 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
I20240503 07:59:27.111076 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
W20240503 07:59:27.117175 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.145772 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.204242 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.207432 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.212152 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212328 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212409 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212575 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212622 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212853 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.214320 14383 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.214679 14363 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:27.214815 14364 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:27.215545 14383 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.215546 14363 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.215641 14364 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.216523 14364 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:27.216604 14364 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.216900 14383 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.216943 14363 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:27.216979 14383 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.217013 14363 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.217075 14364 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217164 14364 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217193 14364 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217294 14364 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217387 14383 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217429 14363 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217541 14363 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217546 14364 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.217574 14363 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217466 14383 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217621 14383 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217676 14363 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217753 14383 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217948 14363 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.218020 14383 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.284035 14197 consensus_queue.cc:1035] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.286871 14197 consensus_queue.cc:1035] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.313598 14243 consensus_queue.cc:1035] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.316380 13931 consensus_queue.cc:1035] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.340521 14335 raft_consensus.cc:482] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.340610 14335 raft_consensus.cc:504] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.341017 14335 leader_election.cc:288] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.341275 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:27.341431 6430 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:27.341691 6235 leader_election.cc:302] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:27.341720 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:27.341823 14335 raft_consensus.cc:2781] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.341873 14335 raft_consensus.cc:482] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.341907 14335 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.341889 8975 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:27.342895 14335 raft_consensus.cc:504] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.343209 14335 leader_election.cc:288] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.343484 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:27.343575 8975 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.343695 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:27.343798 6430 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.344470 8975 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:27.344614 6430 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:27.344797 6234 leader_election.cc:302] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:27.344950 14335 raft_consensus.cc:2781] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.345094 14335 raft_consensus.cc:686] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:27.345250 14335 consensus_queue.cc:237] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.346992 6156 catalog_manager.cc:5484] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:27.364709 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:27.370169 6670 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.370447 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.372280 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.372522 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.372676 14396 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.372807 8957 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.373024 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.373582 14396 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.374179 14383 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.375087 14383 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.375399 8822 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.375416 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.375695 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.375859 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.376157 6539 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.376444 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.376492 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.376689 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.378187 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.378410 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.378638 14400 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.378800 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.379011 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.379527 14383 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.379602 14383 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:27.379649 14400 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.380057 14383 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.380147 14383 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.380177 14383 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.380281 14383 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.380563 14383 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.380729 14383 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.380833 14396 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.380905 14396 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:27.381299 14396 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.381449 14396 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.381510 14396 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.381573 14383 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.381928 14400 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.381915 14396 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.382043 14400 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.383354 14383 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.383435 14383 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.383466 14400 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.383559 14396 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.383566 14400 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.383598 14400 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.383688 14400 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.383744 14396 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.383984 14400 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.384013 14383 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.384101 14383 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.384132 14383 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.384142 14400 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.384244 14383 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.384500 14383 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.385038 14400 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.385922 14396 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.386540 14399 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:27.386852 14396 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.386950 14396 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.387423 14396 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.389143 14396 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.389219 14396 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.389348 14396 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.389604 14204 consensus_queue.cc:1035] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.389688 14396 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.389827 14399 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.391691 14399 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:27.391798 14399 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:27.392221 14399 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.392390 14399 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.392449 14399 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.392598 14399 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.392937 14399 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.393085 14400 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.393107 14399 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:27.393193 14400 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:27.393649 14400 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.393772 14400 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.393826 14400 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.393973 14400 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.394088 14399 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.394447 14400 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.394703 14400 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.395735 14400 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.397153 14400 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.397234 14400 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.397730 14400 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.397812 14400 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.397840 14400 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.397953 14400 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.398204 14400 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.398849 14399 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:27.398922 14399 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:27.399329 14399 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.399433 14399 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.399484 14399 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.399614 14399 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.399865 14399 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.400609 14287 consensus_queue.cc:1035] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:27.402832 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.440981 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:27.448565 13931 raft_consensus.cc:482] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.448680 13931 raft_consensus.cc:504] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.449102 13931 leader_election.cc:288] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.449662 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:27.449810 6690 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.450120 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:27.450273 8975 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.450264 6496 leader_election.cc:302] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.450436 13931 raft_consensus.cc:2781] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.450503 13931 raft_consensus.cc:482] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.450537 13931 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.451562 13931 raft_consensus.cc:504] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.451848 13931 leader_election.cc:288] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.452216 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:27.452307 8975 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.452307 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:27.452407 6690 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.453220 8975 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.453231 6690 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.453501 6496 leader_election.cc:302] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.453655 13931 raft_consensus.cc:2781] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.453742 13931 raft_consensus.cc:686] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:27.453892 13931 consensus_queue.cc:237] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.455924 6148 catalog_manager.cc:5484] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:27.465571 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:27.469017 8975 raft_consensus.cc:1261] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.469112 6430 raft_consensus.cc:1261] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.469429 14394 consensus_queue.cc:1035] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.469595 14335 consensus_queue.cc:1035] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.472429 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.472708 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
I20240503 07:59:27.472833 14416 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:27.473040 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.473415 14417 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:27.473783 6156 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
I20240503 07:59:27.474309 14392 mvcc.cc:205] Tried to move back new op lower bound from 7023506093949599744 to 7023506093447274496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506093949599744}]
I20240503 07:59:27.478390 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.478796 14418 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:27.479373 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 1)
I20240503 07:59:27.479491 6154 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
W20240503 07:59:27.519277 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.540104 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.557934 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.585453 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:27.590327 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.590970 14416 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:27.591161 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.591590 14418 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:27.591913 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.592190 14417 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:27.595327 13931 raft_consensus.cc:482] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.595428 13931 raft_consensus.cc:504] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.598647 13931 leader_election.cc:288] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.599016 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:27.599148 6690 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.599162 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 2)
I20240503 07:59:27.599503 6496 leader_election.cc:302] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.599673 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:27.599771 13931 raft_consensus.cc:2781] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.599797 8842 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.599823 13931 raft_consensus.cc:482] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.599859 13931 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.601137 13931 raft_consensus.cc:504] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.601670 13931 leader_election.cc:288] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.602034 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:27.602128 6690 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.604471 6690 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.604935 6496 leader_election.cc:302] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.605288 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:27.605388 8842 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.606253 8842 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.606776 13931 raft_consensus.cc:2781] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.606796 14334 raft_consensus.cc:482] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.606885 14334 raft_consensus.cc:504] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.606948 13931 raft_consensus.cc:686] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:27.607077 13931 consensus_queue.cc:237] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.607213 14334 leader_election.cc:288] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.607601 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:27.607591 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:27.607702 6560 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:27.607702 8842 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:27.607960 8912 leader_election.cc:302] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:27.608152 14334 raft_consensus.cc:2781] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.608206 14334 raft_consensus.cc:482] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.608248 14334 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.609074 14334 raft_consensus.cc:504] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.609378 14334 leader_election.cc:288] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.609328 6154 catalog_manager.cc:5484] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:27.609658 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:27.609717 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:27.609750 6560 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.609794 8842 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.610567 8842 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:27.610567 6560 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:27.610863 8910 leader_election.cc:302] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.611128 14334 raft_consensus.cc:2781] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.611267 14334 raft_consensus.cc:686] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:27.611407 14334 consensus_queue.cc:237] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.613997 6148 catalog_manager.cc:5484] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:27.629859 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.630478 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.662199 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:27.664083 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.674923 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.701126 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:27.706084 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.706624 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.706624 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.710902 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 3)
W20240503 07:59:27.753296 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.806010 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.813761 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.813913 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.816037 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.816617 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:27.816933 14423 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:27.816915 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:27.817075 14423 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.817173 14424 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:27.817247 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:27.817322 14424 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:27.817507 14425 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:27.817562 14424 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.817591 14423 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.817663 14425 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.817829 14425 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.818109 14424 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.818353 14423 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.818571 14425 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.820676 14423 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820688 14425 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820698 14424 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820760 14424 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.820807 14423 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.820816 14425 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.821090 14425 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:27.821197 14423 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:27.821247 14424 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:27.821740 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
I20240503 07:59:27.821820 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
I20240503 07:59:27.821995 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
W20240503 07:59:27.852015 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:27.860857 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:27.893347 14243 consensus_queue.cc:1035] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.896653 14243 consensus_queue.cc:1035] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:27.910600 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.918206 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.922830 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.922925 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.923086 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.923148 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.923125 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.923358 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.925422 14434 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:27.925505 14432 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.925642 14433 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:27.926492 14432 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927086 14434 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927687 14432 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.927819 14433 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927923 14432 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.928467 14432 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.928655 14432 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.928742 14432 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.928958 14432 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.929250 14432 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.929693 14433 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:27.929808 14434 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:27.929884 14434 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:27.930295 14434 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.930419 14434 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.930471 14434 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.930593 14434 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.931107 14434 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.929777 14433 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:27.931560 14433 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.931655 14433 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.931691 14433 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.931805 14433 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.932829 14433 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:27.992044 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:28.041558 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.113791 13931 consensus_queue.cc:1035] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:28.114255 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:28.118261 13931 consensus_queue.cc:1035] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.122133 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:28.128211 8957 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128485 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.128540 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128540 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128868 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.129187 6408 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.129429 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.130415 14442 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.130566 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.130774 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.131457 14442 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.131477 14433 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.131722 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.132434 14433 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.132951 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.133217 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.133669 14433 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.133750 14433 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:28.134300 14433 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.134423 14433 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.134457 14433 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.134478 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.134579 14433 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.134682 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.134956 14434 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.134958 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.135205 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.135399 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.135608 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.135900 14434 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.137445 14432 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:28.137882 14446 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:28.138384 14445 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:28.138407 14432 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.139256 14446 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.139991 14442 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.141602 14442 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:28.141969 14433 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:59:28.142060 14442 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.142161 14442 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.142246 14442 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.142395 14442 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.142704 14432 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:28.142781 14432 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.142987 14446 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.143077 14446 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.143170 14432 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143262 14432 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.143292 14432 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.143389 14432 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143657 14432 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.143653 14442 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.143808 14442 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.143877 14446 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143985 14446 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.144019 14446 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.144155 14446 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.144539 14446 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.146103 14446 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
W20240503 07:59:28.147189 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.148267 14442 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.149899 14445 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.150848 14440 consensus_queue.cc:1035] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.152442 14442 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.152535 14442 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:28.148324 14446 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.152978 14442 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.153070 14442 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.153101 14442 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.153220 14442 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.153486 14442 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.153973 14446 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.154044 14446 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:28.154419 14446 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.154503 14446 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.154529 14446 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.154629 14446 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.154886 14446 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.157104 14434 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.157191 14434 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:28.157594 14434 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.157685 14434 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.157709 14434 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.157802 14434 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.158053 14434 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.158200 14434 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.159014 14434 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.159986 14334 consensus_queue.cc:1035] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.164151 14434 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.164233 14434 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:28.164999 14434 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.165123 14434 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.165207 14434 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.165319 14434 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.165586 14434 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.169054 14445 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:28.169137 14445 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 07:59:28.169782 14445 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.169884 14445 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.169957 14445 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.170094 14445 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.170372 14445 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.190119 14314 raft_consensus.cc:482] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.190182 14427 raft_consensus.cc:482] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.190209 14314 raft_consensus.cc:504] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.190265 14427 raft_consensus.cc:504] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.190563 14314 leader_election.cc:288] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.190636 14427 leader_election.cc:288] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:28.190821 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.190929 8581 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.190922 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.190914 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.191016 6296 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.191016 8975 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.191134 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:28.191177 6625 leader_election.cc:302] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.191244 6430 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.191256 8516 leader_election.cc:302] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.191313 14427 raft_consensus.cc:2781] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.191356 14427 raft_consensus.cc:482] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.191407 14427 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.191496 14314 raft_consensus.cc:2781] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.191545 14314 raft_consensus.cc:482] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.191592 14314 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.192332 14427 raft_consensus.cc:504] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.192461 14314 raft_consensus.cc:504] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.192658 14427 leader_election.cc:288] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:28.192790 14314 leader_election.cc:288] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.193066 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:28.193096 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.193148 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.193183 8581 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193192 6430 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193241 8975 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193907 8581 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.194104 8975 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.194104 6430 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.194149 6625 leader_election.cc:302] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.194288 14427 raft_consensus.cc:2781] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.194312 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.194361 14427 raft_consensus.cc:686] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:28.194387 8516 leader_election.cc:302] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.194432 6296 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.194528 14314 raft_consensus.cc:2781] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.194500 14427 consensus_queue.cc:237] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.194705 14314 raft_consensus.cc:686] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.194826 14314 consensus_queue.cc:237] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.195344 6296 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.196837 6155 catalog_manager.cc:5484] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:28.197237 6148 catalog_manager.cc:5484] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.202980 14427 raft_consensus.cc:482] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.203058 14427 raft_consensus.cc:504] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.203379 14427 leader_election.cc:288] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.203611 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.203645 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.203718 6296 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.203738 8581 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.203949 6625 leader_election.cc:302] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.204103 14427 raft_consensus.cc:2781] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.204164 14427 raft_consensus.cc:482] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.204195 14427 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.204905 14427 raft_consensus.cc:504] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.205212 14427 leader_election.cc:288] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.205468 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.205564 6296 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.205569 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.205643 8581 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.206360 8581 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.206360 6296 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.206673 6625 leader_election.cc:302] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.206950 14427 raft_consensus.cc:2781] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.207023 14427 raft_consensus.cc:686] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:28.207136 14427 consensus_queue.cc:237] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.209296 6148 catalog_manager.cc:5484] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.264884 14204 raft_consensus.cc:482] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.264977 14204 raft_consensus.cc:504] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.265416 14204 leader_election.cc:288] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.265657 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.265777 8975 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.265863 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:28.265990 8842 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.266036 6365 leader_election.cc:302] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.266312 14204 raft_consensus.cc:2781] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.266364 14204 raft_consensus.cc:482] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.266415 14204 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.267156 14204 raft_consensus.cc:504] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.267422 14204 leader_election.cc:288] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.267688 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.267727 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:28.267791 8975 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.267819 8842 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.268594 8842 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.268606 8975 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.268851 6365 leader_election.cc:302] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.268994 14204 raft_consensus.cc:2781] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.269079 14204 raft_consensus.cc:686] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:28.269217 14204 consensus_queue.cc:237] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.271042 6148 catalog_manager.cc:5484] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:28.313283 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:28.317072 8581 raft_consensus.cc:1261] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.317193 6430 raft_consensus.cc:1261] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.317592 14426 consensus_queue.cc:1035] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.317970 14426 consensus_queue.cc:1035] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.320807 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.321288 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.323559 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.324326 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.325902 14471 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.327626 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 1)
I20240503 07:59:28.328424 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.329134 14470 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.329989 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.330193 14466 mvcc.cc:205] Tried to move back new op lower bound from 7023506097423388672 to 7023506096926064640. Current Snapshot: MvccSnapshot[applied={T|T < 7023506097423388672}]
I20240503 07:59:28.330344 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.330651 14472 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:28.333117 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.433784 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:28.439113 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.439442 14470 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.439518 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.439790 14471 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.440182 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.440677 14472 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:28.444540 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 2)
I20240503 07:59:28.547570 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:28.552691 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.553203 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.553718 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.558426 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 3)
W20240503 07:59:28.595721 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.616461 14460 consensus_queue.cc:1035] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.623111 14460 consensus_queue.cc:1035] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.626332 14388 consensus_queue.cc:1035] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.629024 14427 consensus_queue.cc:1035] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.660571 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.660718 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.662617 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.663144 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:28.663167 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:28.663256 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:28.663468 14481 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:28.663522 14482 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:28.663596 14483 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:28.663630 14481 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.663636 14482 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.663736 14483 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:28.663749 14482 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.663790 14481 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.664034 14483 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.664371 14482 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.664371 14481 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.664553 14483 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.666708 14481 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666708 14482 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666734 14483 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666780 14482 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.666800 14483 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.666922 14481 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.667055 14482 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:28.667098 14483 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:28.667495 14481 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:28.667788 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
I20240503 07:59:28.667894 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
I20240503 07:59:28.668330 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
W20240503 07:59:28.759691 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.764277 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:28.769160 8559 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.769270 6540 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.769430 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.769497 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.771260 14487 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.771478 14486 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.771553 6670 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.771771 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:28.772208 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd4400894d14f839670d209468f1929: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.772429 14486 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.773561 14486 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.773648 14486 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.773942 14491 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:28.774044 14486 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.774150 14486 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.774183 14486 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.774291 14486 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.774576 14486 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.774945 14491 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.774945 14487 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.776171 14487 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.776209 14491 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:28.776289 14487 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.776372 14491 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.776803 14487 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.776935 14487 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.776986 14487 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.777045 14491 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777101 14487 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777145 14491 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.777189 14491 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.777302 14491 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777380 14487 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.777597 14491 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:28.779444 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.798072 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:28.803248 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803372 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803539 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.803678 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.803701 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803926 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.805022 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.805197 14487 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.805269 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.805686 6277 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.805951 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.806169 14487 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.806535 8957 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.806676 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.806771 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.806914 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.808246 14495 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:28.808954 6540 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.809090 14486 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.809089 14496 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.809154 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.809828 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.809970 14496 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.810041 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.810434 14486 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.812110 14486 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.812160 14487 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.812192 14486 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:28.812206 14495 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.812218 14487 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:28.812585 14487 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.812605 14486 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.812683 14486 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.812709 14486 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.812739 14487 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.812796 14487 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.812902 14487 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.813043 14486 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.813313 14486 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.813452 14486 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.813493 14497 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:28.813958 14487 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.814106 14487 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.814237 14486 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.814417 14494 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.814915 14487 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815239 14494 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815702 14497 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815673 14204 consensus_queue.cc:1035] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.815856 14487 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.815927 14487 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.816289 14487 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.816377 14487 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.816408 14487 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.816517 14487 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.816605 14495 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.816668 14495 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:28.816751 14487 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.817034 14495 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.817114 14495 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.817138 14495 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.817230 14495 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.817453 14495 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.821820 14486 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.821895 14486 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:28.822100 14494 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.822170 14494 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:28.822304 14486 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.822412 14486 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.822439 14486 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.822541 14486 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.822722 14494 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.822789 14486 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.822814 14494 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.822844 14494 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.824024 14494 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.824415 14494 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.824610 14494 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.825549 14494 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.826421 14494 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.826543 14494 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.826939 14494 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.827864 14494 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.827913 14494 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.828032 14494 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.828279 14494 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.828799 14497 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:28.828879 14497 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:28.828994 14496 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.829078 14496 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:28.829275 14497 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.829375 14497 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.829402 14497 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.829515 14496 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.829619 14496 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.829653 14496 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.829754 14496 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.830006 14496 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.831568 14497 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.832430 14497 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:28.838491 14204 consensus_queue.cc:1035] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.844486 14475 raft_consensus.cc:482] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.844595 14475 raft_consensus.cc:504] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.845005 14475 leader_election.cc:288] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:28.845291 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:28.845430 6560 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.845500 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:28.845615 6690 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.845690 8514 leader_election.cc:302] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.845836 14475 raft_consensus.cc:2781] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.845886 14475 raft_consensus.cc:482] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.845928 14475 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.846892 14475 raft_consensus.cc:504] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.847208 14475 leader_election.cc:288] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:28.847462 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:28.847539 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:28.847579 6690 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.847648 6560 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.848284 6690 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.848502 6560 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.848556 8516 leader_election.cc:302] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.848696 14475 raft_consensus.cc:2781] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.848817 14475 raft_consensus.cc:686] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.848935 14475 consensus_queue.cc:237] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.850958 6154 catalog_manager.cc:5484] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.898754 14204 raft_consensus.cc:482] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.898859 14204 raft_consensus.cc:504] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.899188 14204 leader_election.cc:288] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.899494 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:28.899629 6560 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.899601 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.899713 8575 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.899889 6363 leader_election.cc:302] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.900055 14204 raft_consensus.cc:2781] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.900110 14204 raft_consensus.cc:482] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.900136 14204 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.901054 14204 raft_consensus.cc:504] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.901403 14204 leader_election.cc:288] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.901666 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:28.901702 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.901773 6560 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.901801 8575 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.902683 8575 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.902714 6560 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.902916 6363 leader_election.cc:302] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.903079 14204 raft_consensus.cc:2781] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.903177 14204 raft_consensus.cc:686] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:28.903299 14204 consensus_queue.cc:237] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.905288 6154 catalog_manager.cc:5484] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:28.922616 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:28.937292 6346 tablet.cc:2369] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.937682 14243 raft_consensus.cc:482] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.937764 14243 raft_consensus.cc:504] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.938129 14243 leader_election.cc:288] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.938364 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.938477 6296 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:28.938719 6494 leader_election.cc:302] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.938755 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.938871 8976 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:28.938875 14243 raft_consensus.cc:2781] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.938926 14243 raft_consensus.cc:482] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.938958 14243 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.939818 14243 raft_consensus.cc:504] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.940091 14243 leader_election.cc:288] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.940423 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.940471 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.940528 6296 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.940569 8976 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.941315 8976 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:28.941365 6296 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:28.941572 6496 leader_election.cc:302] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:28.941722 14243 raft_consensus.cc:2781] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.941790 14243 raft_consensus.cc:686] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:28.941929 14243 consensus_queue.cc:237] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.943953 6154 catalog_manager.cc:5484] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:28.949069 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.961504 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:28.964726 6690 raft_consensus.cc:1261] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.965116 6560 raft_consensus.cc:1261] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.965174 14460 consensus_queue.cc:1035] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.965596 14460 consensus_queue.cc:1035] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.965880 14460 raft_consensus.cc:482] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.965955 14460 raft_consensus.cc:504] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.966392 14460 leader_election.cc:288] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.966730 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:28.966867 6429 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.967156 8517 leader_election.cc:302] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.967558 14475 raft_consensus.cc:2781] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.967614 14475 raft_consensus.cc:482] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.967650 14475 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.967979 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.968400 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:28.968516 8843 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.968576 14475 raft_consensus.cc:504] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.968719 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.968904 14475 leader_election.cc:288] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.969019 14471 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.970364 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.970772 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.970973 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.971074 14470 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.971283 14518 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:28.971657 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:28.971765 8843 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.972223 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.972678 8843 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.973014 8516 leader_election.cc:302] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.973176 14475 raft_consensus.cc:2781] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.973258 14475 raft_consensus.cc:686] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.973385 14475 consensus_queue.cc:237] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.973416 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:28.973506 6429 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.974417 6429 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.974777 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.977245 6150 catalog_manager.cc:5484] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.978574 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 1)
W20240503 07:59:29.001771 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.029019 6608 tablet.cc:2369] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:29.036656 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.038589 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.082777 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.088330 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.088666 14471 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:29.088801 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.089097 14518 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.091609 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.091909 14470 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.094230 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 2)
W20240503 07:59:29.095244 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.106216 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:29.183944 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.196058 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:29.201210 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.201762 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.201819 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:29.207600 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.208379 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 3)
W20240503 07:59:29.269780 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.308950 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.309079 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.311028 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.311560 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:29.311654 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:29.311839 14524 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:29.311965 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:29.312060 14524 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.312106 14525 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:29.312237 14526 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:29.312248 14524 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312274 14525 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.312389 14526 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.312474 14525 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312553 14526 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312847 14524 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.313082 14526 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.313078 14525 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.315052 14524 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315124 14526 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315124 14525 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315162 14524 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315196 14525 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315316 14526 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315411 14524 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:29.315497 14525 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:29.315630 14526 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:29.316071 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.316234 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.316648 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.333252 14522 consensus_queue.cc:1035] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.336436 14508 consensus_queue.cc:1035] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.355346 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.412719 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:29.417976 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.418227 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.418378 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.418603 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.419986 14529 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:29.420238 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.420327 14530 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.420449 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.421403 14530 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.422390 14529 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.422613 14532 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.423478 14529 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:29.423501 14532 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.423542 14529 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:29.423934 14529 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.424064 14529 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.424090 14529 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.424196 14529 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.424474 14529 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.425647 14532 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.425767 14532 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.426190 14532 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.426286 14532 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.426316 14532 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.426465 14532 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.430016 14532 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:29.433341 14530 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.433437 14530 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:29.433835 14530 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.433938 14530 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.433974 14530 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.434103 14530 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.434371 14530 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.435289 14475 consensus_queue.cc:1035] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.439515 14517 consensus_queue.cc:1035] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.440456 6739 tablet.cc:2369] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:29.463778 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.465698 14243 consensus_queue.cc:1035] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.469139 14356 consensus_queue.cc:1035] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.515164 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:29.520664 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521065 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.521224 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521461 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.521414 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521667 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.523082 14532 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.523319 8822 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.523599 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.523947 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.524066 14532 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.524187 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.525693 14532 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.525775 14532 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.526154 14544 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:29.526129 6278 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.526208 14532 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.526299 14532 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.526329 14532 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.526345 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.526454 6279 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.526449 14532 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.526628 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.526737 14532 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.527312 14544 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.528208 14532 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.528507 6408 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.528690 6410 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.528759 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.528941 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.530683 14530 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.531543 14530 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.531769 14542 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:29.533136 14542 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.534217 14542 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:29.534350 14542 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.534488 14532 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.534718 14542 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.534827 14542 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.534858 14542 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.534986 14542 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.535331 14542 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.535431 14532 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.535511 14532 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:29.535825 14544 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:29.535868 14532 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.535950 14532 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.535981 14532 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.536011 14544 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:29.536309 14532 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.536517 14544 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.536638 14544 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.536667 14544 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.536757 14544 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.535097 14545 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.536898 14532 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.537184 14544 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.537782 14545 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.539252 14530 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.539326 14530 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:29.539731 14530 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.539827 14530 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.539868 14530 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.539966 14530 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.539311 14545 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.540227 14530 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.540406 14530 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.541419 14530 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.542665 14545 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:29.543141 14545 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.543248 14545 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.543279 14545 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.543388 14545 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.543648 14545 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.543879 14545 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.544760 14530 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.544873 14530 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:29.544982 14545 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.545303 14530 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.545395 14530 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.545428 14530 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.545543 14530 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.545763 14530 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.546083 14545 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.546172 14545 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.546603 14545 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.546685 14545 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.546715 14545 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.546969 14545 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.547362 14545 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.547546 14545 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.548377 14545 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.549391 14545 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.549494 14545 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:29.549906 14545 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.550009 14545 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.550086 14545 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.550244 14545 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.550555 14545 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.555344 14523 raft_consensus.cc:482] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.555429 14523 raft_consensus.cc:504] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.555825 14523 leader_election.cc:288] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:29.556072 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:29.556144 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:29.556174 6296 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.556258 8575 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.556424 6625 leader_election.cc:302] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:29.556572 14523 raft_consensus.cc:2781] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.556628 14523 raft_consensus.cc:482] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.556664 14523 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.557509 14523 raft_consensus.cc:504] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.557794 14523 leader_election.cc:288] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:29.558038 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:29.558121 6296 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.558117 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:29.558218 8575 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.559010 8575 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.559068 6296 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.559314 6625 leader_election.cc:302] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:29.559463 14523 raft_consensus.cc:2781] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.559546 14523 raft_consensus.cc:686] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:29.559679 14523 consensus_queue.cc:237] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.561522 6150 catalog_manager.cc:5484] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:29.587332 14335 raft_consensus.cc:482] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.587424 14335 raft_consensus.cc:504] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.587811 14335 leader_election.cc:288] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:29.588079 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:29.588099 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:29.588207 6690 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:29.588207 6429 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:29.588461 6235 leader_election.cc:302] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.588616 14335 raft_consensus.cc:2781] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.588662 14335 raft_consensus.cc:482] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.588696 14335 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.589491 14335 raft_consensus.cc:504] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.589787 14335 leader_election.cc:288] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:29.590049 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:29.590040 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:29.590145 6429 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.590145 6690 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.591006 6690 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:29.591006 6429 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:29.591266 6235 leader_election.cc:302] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.591429 14335 raft_consensus.cc:2781] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.591508 14335 raft_consensus.cc:686] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:29.591640 14335 consensus_queue.cc:237] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.593873 6150 catalog_manager.cc:5484] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:29.616708 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.625414 14523 raft_consensus.cc:482] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.625504 14523 raft_consensus.cc:504] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.625881 14523 leader_election.cc:288] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:29.626121 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:29.626160 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:29.626250 6560 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.626266 6429 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.626557 6625 leader_election.cc:302] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:29.626680 14523 raft_consensus.cc:2781] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.626731 14523 raft_consensus.cc:482] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.626773 14523 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.627624 14523 raft_consensus.cc:504] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.627933 14523 leader_election.cc:288] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:29.628176 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:29.628253 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:29.628278 6560 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.628347 6429 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.629191 6429 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.629220 6560 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.629520 6625 leader_election.cc:302] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:29.630009 14523 raft_consensus.cc:2781] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.630088 14523 raft_consensus.cc:686] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:29.630203 14523 consensus_queue.cc:237] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
W20240503 07:59:29.631373 6477 tablet.cc:2369] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:29.632272 6150 catalog_manager.cc:5484] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:29.692864 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:29.754390 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.764444 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:29.768700 6429 raft_consensus.cc:1261] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.768700 6560 raft_consensus.cc:1261] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.769065 14427 consensus_queue.cc:1035] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.769201 14523 consensus_queue.cc:1035] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.771255 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.774173 6156 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.774224 14508 raft_consensus.cc:482] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.774296 14508 raft_consensus.cc:504] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.774178 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.774569 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.774632 14508 leader_election.cc:288] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:29.774847 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.774899 14565 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.774987 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:29.775107 8842 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:29.775215 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.775383 6365 leader_election.cc:302] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.775771 14508 raft_consensus.cc:2781] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.775820 14508 raft_consensus.cc:482] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.775862 14508 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.775925 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:29.776039 6296 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:29.777153 14508 raft_consensus.cc:504] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.777390 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.777468 14508 leader_election.cc:288] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:29.777550 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.777712 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.777815 14563 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:29.778097 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:29.778205 8842 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.779255 8842 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:29.779680 6365 leader_election.cc:302] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.779981 14508 raft_consensus.cc:2781] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.780071 14508 raft_consensus.cc:686] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:29.780105 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:29.780205 6296 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.780207 14508 consensus_queue.cc:237] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.781136 6296 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:29.782219 14562 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.782629 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 1)
I20240503 07:59:29.785281 6148 catalog_manager.cc:5484] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.887091 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.891726 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.892041 14562 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.892289 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.892678 14563 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:29.893712 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.894003 14565 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.896818 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 2)
W20240503 07:59:29.937093 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.939532 14427 consensus_queue.cc:1035] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.942755 14564 consensus_queue.cc:1035] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.981283 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 07:59:29.981863 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.998322 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.003535 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.004472 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:30.004536 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.009733 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 3)
W20240503 07:59:30.010143 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.010526 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:30.025977 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 07:59:30.027217 14335 consensus_queue.cc:1035] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.030653 14335 consensus_queue.cc:1035] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:30.081787 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.109874 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 07:59:30.113297 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.113441 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.115357 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.115940 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:30.115974 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:30.116108 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:30.116519 14577 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:30.116542 14576 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:30.116678 14576 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.116706 14577 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.116843 14576 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.116958 14577 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.117105 14575 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:30.117254 14575 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.117430 14575 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.117550 14577 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.117551 14576 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.117885 14575 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.119773 14575 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119836 14575 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.119835 14577 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119832 14576 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119902 14576 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.119904 14577 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.120240 14577 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:30.120431 14575 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:30.120543 14576 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:30.121068 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
I20240503 07:59:30.121098 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
I20240503 07:59:30.121250 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
W20240503 07:59:30.122293 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 07:59:30.123034 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.138583 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 07:59:30.139135 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.187517 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 07:59:30.206079 14567 consensus_queue.cc:1035] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.208693 14204 consensus_queue.cc:1035] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.217085 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.221664 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.221841 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.221935 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.222061 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.224241 14583 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:30.224367 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.224520 14582 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:30.224596 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.225442 14583 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.225488 14582 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.226857 14582 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:30.226971 14582 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.227105 14586 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.228008 14586 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.228302 14583 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:30.228436 14583 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.229035 14583 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.229192 14583 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.229267 14583 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.229436 14583 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.229761 14583 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.230592 14582 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.230710 14582 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.230752 14582 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.230873 14582 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.231232 14582 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:30.232271 14586 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.232347 14586 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:30.232656 14586 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.232749 14586 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.232841 14586 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.232960 14586 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.233227 14586 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 07:59:30.234864 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.266206 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.283321 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.322376 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:30.328003 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.328290 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.328477 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.328723 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.328963 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.329208 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.331048 14588 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:30.332090 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.332113 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.332329 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.332329 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.333309 8823 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.333571 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.334422 14582 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:30.334527 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.334743 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.335281 14582 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.335517 14589 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:30.333303 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.335887 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.336844 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.336943 14589 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.337103 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.337947 14589 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:30.338021 14589 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:30.338399 14589 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.338495 14589 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.338522 14589 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.338620 14589 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.338876 14589 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.339082 14586 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.339191 14588 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.339850 14586 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.340219 14588 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:30.340289 14588 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:30.340698 14588 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.340837 14588 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.340893 14588 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.341022 14588 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.341346 14588 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.341506 14588 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:30.342072 14590 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:30.342414 14588 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.343155 14590 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.343708 14582 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:30.343788 14582 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:30.343992 14586 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.344066 14586 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:30.344197 14582 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.344286 14582 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344316 14582 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.344372 14590 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:30.344439 14590 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.344431 14582 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.344504 14586 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344594 14586 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344623 14586 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.344695 14582 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.344722 14586 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344863 14590 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344986 14590 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344987 14586 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.345041 14590 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.345177 14590 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.345544 14590 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.346360 14591 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.347333 14591 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.348553 14588 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:30.348627 14588 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:30.349017 14591 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.349076 14588 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.349148 14591 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:30.349161 14588 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.349189 14588 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.349292 14588 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.349557 14588 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.349623 14591 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.349709 14591 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.349754 14591 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.349869 14591 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.350211 14591 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.350467 14591 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.352270 14591 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.354588 14335 raft_consensus.cc:482] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.354686 14335 raft_consensus.cc:504] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.355136 14335 leader_election.cc:288] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.355157 14591 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.355226 14591 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:30.355406 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:30.355473 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:30.355511 8575 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:30.355579 8976 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:30.355638 14591 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.355751 14591 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.355791 14591 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.355820 6232 leader_election.cc:302] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:30.355921 14591 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.356128 14335 raft_consensus.cc:2781] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.356181 14335 raft_consensus.cc:482] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.356232 14335 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.356492 14591 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.356628 14591 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.357123 14335 raft_consensus.cc:504] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.357419 14335 leader_election.cc:288] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.357688 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:30.357803 8575 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.357950 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:30.358034 8976 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.358405 14591 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.358687 8575 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:30.358834 8976 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:30.359088 6232 leader_election.cc:302] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:30.359580 14473 raft_consensus.cc:2781] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.359668 14473 raft_consensus.cc:686] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:30.359777 14473 consensus_queue.cc:237] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.360092 14591 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.360306 14591 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
W20240503 07:59:30.360837 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 07:59:30.361799 14591 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.361771 6154 catalog_manager.cc:5484] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.361876 14591 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.361902 14591 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.362007 14591 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.362263 14591 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:30.379738 14564 raft_consensus.cc:482] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.379837 14564 raft_consensus.cc:504] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.380209 14564 leader_election.cc:288] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:30.380460 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:30.380488 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:30.380561 8842 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:30.380587 6429 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:30.380808 6627 leader_election.cc:302] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:30.380982 14564 raft_consensus.cc:2781] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.381037 14564 raft_consensus.cc:482] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.381062 14564 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.381906 14564 raft_consensus.cc:504] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.382230 14564 leader_election.cc:288] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:30.382498 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:30.382508 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:30.382596 6429 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.382618 8842 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.383440 8842 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:30.383440 6429 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:30.383678 6627 leader_election.cc:302] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:30.383829 14564 raft_consensus.cc:2781] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.383915 14564 raft_consensus.cc:686] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:30.384124 14564 consensus_queue.cc:237] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.386427 6156 catalog_manager.cc:5484] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.418143 14455 raft_consensus.cc:482] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.418231 14455 raft_consensus.cc:504] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.418628 14455 leader_election.cc:288] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:30.418886 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:30.418903 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:30.418989 6429 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:30.418987 8843 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:30.419220 8913 leader_election.cc:302] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:30.419380 14455 raft_consensus.cc:2781] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.419431 14455 raft_consensus.cc:482] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.419456 14455 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.420321 14455 raft_consensus.cc:504] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.420712 14455 leader_election.cc:288] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:30.420970 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:30.421077 8843 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.421142 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:30.421243 6430 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.422044 8843 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:30.422165 6430 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:30.422302 8912 leader_election.cc:302] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:30.422521 14455 raft_consensus.cc:2781] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.422629 14455 raft_consensus.cc:686] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:30.422766 14455 consensus_queue.cc:237] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.425107 6156 catalog_manager.cc:5484] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:30.438886 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.447893 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:30.454813 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.468209 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:30.477031 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:30.480540 8575 raft_consensus.cc:1261] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.480819 8976 raft_consensus.cc:1261] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.480916 14335 consensus_queue.cc:1035] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.481175 14601 consensus_queue.cc:1035] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.483469 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.483790 14610 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:30.484802 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.485088 14611 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:30.485658 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.485996 6154 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.486358 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.489797 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.490132 14612 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:30.490492 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 1)
I20240503 07:59:30.493191 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
W20240503 07:59:30.517398 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 07:59:30.518013 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:30.536352 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.577417 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.587909 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.595669 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:30.601086 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.601576 14610 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:30.601985 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.602355 14611 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:30.607026 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.607285 14612 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:30.607753 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 2)
W20240503 07:59:30.639019 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.671947 14291 raft_consensus.cc:482] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.672053 14291 raft_consensus.cc:504] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.672452 14291 leader_election.cc:288] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.672710 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:30.672739 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:30.672871 6560 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:30.672889 8575 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:30.673156 8776 leader_election.cc:302] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:30.673326 14291 raft_consensus.cc:2781] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.673380 14291 raft_consensus.cc:482] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.673424 14291 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.674185 14291 raft_consensus.cc:504] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.674456 14291 leader_election.cc:288] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.674739 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:30.674759 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:30.674834 6560 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.674844 8575 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.675530 6560 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:30.675540 8575 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:30.675772 8776 leader_election.cc:302] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:30.675913 14291 raft_consensus.cc:2781] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.675998 14291 raft_consensus.cc:686] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:30.676124 14291 consensus_queue.cc:237] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.678241 6148 catalog_manager.cc:5484] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.708830 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.713019 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.713516 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.713666 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.717916 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 3)
I20240503 07:59:30.797247 14427 consensus_queue.cc:1035] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.799883 14523 consensus_queue.cc:1035] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.820238 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.820389 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.822582 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.823053 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:30.823215 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:30.823354 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:30.823557 14622 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:30.823662 14624 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:30.823661 14623 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:30.823787 14622 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.823801 14623 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.823868 14624 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.823984 14623 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824023 14624 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824066 14622 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824684 14624 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.824903 14622 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.824985 14623 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.826877 14623 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.826987 14623 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.827263 14622 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.827414 14622 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.827723 14622 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:30.828711 14624 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.828851 14624 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.828904 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.829221 14623 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:30.829258 14624 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:30.829799 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.830472 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.924307 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.929142 8560 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.929395 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.929436 6540 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.929675 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.931239 14625 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.931515 6670 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.931588 14626 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:30.931735 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.932348 14625 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.932425 14626 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.933601 14625 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.933712 14625 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:30.934327 14629 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:30.934427 14626 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:30.934520 14626 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:30.934438 14625 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.934702 14625 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.934758 14625 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.934914 14625 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.934971 14626 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.935057 14626 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.935089 14626 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.935205 14629 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.935209 14626 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.935488 14626 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.936545 14625 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:30.937696 14629 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:30.937778 14629 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.938189 14629 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.938421 14629 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.939535 14629 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.939700 14629 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.940055 14629 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.968590 14455 consensus_queue.cc:1035] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.972138 14632 consensus_queue.cc:1035] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:30.999528 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:31.082301 14291 consensus_queue.cc:1035] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.086320 14291 consensus_queue.cc:1035] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.142741 14523 raft_consensus.cc:482] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.142832 14523 raft_consensus.cc:504] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.143141 14523 leader_election.cc:288] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.143429 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:31.143486 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.143582 8575 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.143594 6560 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.143839 6625 leader_election.cc:302] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.143997 14523 raft_consensus.cc:2781] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.144063 14523 raft_consensus.cc:482] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.144106 14523 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.145028 14523 raft_consensus.cc:504] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.145398 14523 leader_election.cc:288] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.145709 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:31.145825 8575 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.146009 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.146123 6560 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.146651 8575 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.146907 6560 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.146915 6625 leader_election.cc:302] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.147073 14523 raft_consensus.cc:2781] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.147174 14523 raft_consensus.cc:686] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:31.147293 14523 consensus_queue.cc:237] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.149464 6156 catalog_manager.cc:5484] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:31.206988 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:31.212352 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.212478 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.212652 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.212741 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.213524 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.213738 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.214720 14640 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.214946 8955 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.215142 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.215193 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.215451 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.216477 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.216713 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.217123 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.217195 14641 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.217337 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.218077 14641 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.219203 14640 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.219257 14644 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:31.219799 14629 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:31.220301 14640 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.220391 14640 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:31.220645 6540 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.220765 14629 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.220906 14642 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:31.220942 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.221091 14640 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.221197 14640 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.221230 14640 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.221349 14640 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.221644 14640 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.221719 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.221840 14640 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.221933 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.222262 14641 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.222353 14641 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:31.222757 14640 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.222788 14641 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.222887 14641 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.222923 14641 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.223155 14626 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:31.223367 14641 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.223680 14641 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.224035 14626 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.224078 14641 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.225347 14641 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.225916 14644 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.227191 14626 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:31.227279 14626 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:31.227329 14640 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.227398 14640 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:31.227731 14626 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.227833 14626 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.227869 14626 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.227988 14626 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.228070 14642 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.228291 14626 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.228904 14640 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229074 14629 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:31.229112 14640 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.229156 14629 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:31.229158 14640 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.229275 14640 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229519 14642 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:31.229590 14642 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:31.229584 14629 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229672 14629 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.229702 14629 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.229815 14629 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.230062 14629 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.230027 14642 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.230114 14642 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.230144 14642 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.230257 14642 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.230513 14642 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.231830 14640 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:31.232013 14640 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.233608 14640 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.234637 14640 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.234722 14640 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:31.235240 14640 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.235332 14640 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.235360 14640 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.235462 14640 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.235731 14640 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.239686 14644 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:31.239759 14644 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:31.239902 14641 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.239970 14641 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:31.240281 14644 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.240293 14641 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.240370 14644 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.240386 14641 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.240398 14644 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.240409 14641 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.240514 14641 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.240552 14644 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.240746 14641 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.240890 14644 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.261148 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:31.264132 8575 raft_consensus.cc:1261] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.264138 6560 raft_consensus.cc:1261] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.264506 14427 consensus_queue.cc:1035] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.264624 14523 consensus_queue.cc:1035] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.267282 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.267427 14618 mvcc.cc:205] Tried to move back new op lower bound from 7023506109494931456 to 7023506109020590080. Current Snapshot: MvccSnapshot[applied={T|T < 7023506109494931456}]
I20240503 07:59:31.267577 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.267869 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.268023 14656 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:31.268818 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.269073 14657 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.271402 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 1)
I20240503 07:59:31.271597 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.271757 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.271869 14658 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.274926 14632 raft_consensus.cc:482] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.275012 14632 raft_consensus.cc:504] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.275409 14632 leader_election.cc:288] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:31.275655 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.275734 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:31.275777 6298 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:31.275861 6429 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:31.276054 8910 leader_election.cc:302] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:31.276261 14632 raft_consensus.cc:2781] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.276316 14632 raft_consensus.cc:482] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.276360 14632 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.277164 14632 raft_consensus.cc:504] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.277431 14632 leader_election.cc:288] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:31.277690 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.277722 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:31.277776 6298 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.277803 6429 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.278425 6298 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:31.278436 6429 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:31.278683 8913 leader_election.cc:302] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:31.278836 14632 raft_consensus.cc:2781] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.278909 14632 raft_consensus.cc:686] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:31.279022 14632 consensus_queue.cc:237] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.281167 6150 catalog_manager.cc:5484] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.298610 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.319641 14636 raft_consensus.cc:482] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.319736 14636 raft_consensus.cc:504] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.320152 14636 leader_election.cc:288] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.320420 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:31.320473 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.320533 8975 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:31.320593 6560 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:31.320865 8776 leader_election.cc:302] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.321028 14636 raft_consensus.cc:2781] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.321077 14636 raft_consensus.cc:482] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.321111 14636 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.322000 14636 raft_consensus.cc:504] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.322274 14636 leader_election.cc:288] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.322559 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:31.322558 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.322659 8975 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.322674 6560 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.323468 8975 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:31.323575 6560 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:31.323808 8776 leader_election.cc:302] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.324023 14636 raft_consensus.cc:2781] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.324190 14636 raft_consensus.cc:686] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:31.324330 14636 consensus_queue.cc:237] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.326587 6150 catalog_manager.cc:5484] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.362993 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.374382 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:31.378939 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.379304 14656 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:31.379526 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.379807 14658 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.379845 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.380092 14657 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.384510 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 2)
W20240503 07:59:31.423069 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.486069 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:31.490602 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.491501 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.491513 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.497238 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 3)
I20240503 07:59:31.501936 14523 raft_consensus.cc:482] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.502019 14523 raft_consensus.cc:504] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.502400 14523 leader_election.cc:288] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:31.502647 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.502869 6298 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.503155 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:31.503283 8975 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.503566 6627 leader_election.cc:302] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
W20240503 07:59:31.503705 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.503777 14523 raft_consensus.cc:2781] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.503818 14523 raft_consensus.cc:482] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.503860 14523 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.504690 14523 raft_consensus.cc:504] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.505020 14523 leader_election.cc:288] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:31.505273 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:31.505291 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.505367 8975 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.505411 6298 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.506021 8975 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.506116 6298 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.506242 6627 leader_election.cc:302] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:31.506395 14523 raft_consensus.cc:2781] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.506465 14523 raft_consensus.cc:686] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:31.506579 14523 consensus_queue.cc:237] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.508354 6150 catalog_manager.cc:5484] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.517640 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:31.549204 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:31.558929 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:31.579638 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.600298 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.600503 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.602821 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.603338 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:31.603452 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:31.603916 14665 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:31.604048 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:31.604099 14664 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:31.604112 14665 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:31.604228 14664 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.604373 14664 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.604391 14665 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.604915 14665 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.604984 14664 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.606931 14665 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.606941 14664 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.607002 14665 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.607003 14664 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.607252 14664 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:31.607333 14665 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:31.607820 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
I20240503 07:59:31.608050 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
I20240503 07:59:31.608930 14666 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:31.609126 14666 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.609242 14666 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.609881 14666 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.612370 14666 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.612488 14666 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.612807 14666 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:31.613574 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
W20240503 07:59:31.654539 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.674054 14604 consensus_queue.cc:1035] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.676633 14661 consensus_queue.cc:1035] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.704663 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:31.709286 6279 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.709318 6540 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.709528 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.709528 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.711524 14641 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.711834 8560 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.712049 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.712389 14641 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.712929 14626 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:31.713392 14641 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.713470 14641 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:31.713725 14626 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.713894 14641 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.713989 14641 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.714020 14641 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.714102 14674 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:31.714133 14641 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.714383 14641 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.715058 14626 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:31.715138 14626 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:31.715415 14674 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.715507 14626 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.715577 14626 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.715595 14626 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.715687 14626 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.715939 14626 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.717973 14674 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:31.718080 14674 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:31.718509 14674 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.718600 14674 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.718621 14674 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.718711 14674 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.718945 14674 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.733991 14636 consensus_queue.cc:1035] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.736522 14291 consensus_queue.cc:1035] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.788005 14634 raft_consensus.cc:482] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.788107 14634 raft_consensus.cc:504] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.788622 14634 leader_election.cc:288] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
W20240503 07:59:31.789314 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.789363 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.789417 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.789482 6298 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:31.789515 6560 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:31.789737 8514 leader_election.cc:302] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.789935 14634 raft_consensus.cc:2781] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.789999 14634 raft_consensus.cc:482] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.790036 14634 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.790841 14634 raft_consensus.cc:504] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.791144 14634 leader_election.cc:288] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.791438 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.791414 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.791541 6560 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.791550 6298 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.792384 6560 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:31.792394 6298 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:31.792656 8514 leader_election.cc:302] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.792830 14634 raft_consensus.cc:2781] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.792989 14634 raft_consensus.cc:686] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:31.793130 14634 consensus_queue.cc:237] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.795190 6156 catalog_manager.cc:5484] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:31.805665 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:31.838999 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:31.883277 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:31.911718 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:31.915369 6560 raft_consensus.cc:1261] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.915475 6298 raft_consensus.cc:1261] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.915838 14680 consensus_queue.cc:1035] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.916016 14634 consensus_queue.cc:1035] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.918876 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.918888 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.919169 14658 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.919273 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.919544 14657 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.920169 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.920822 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.923856 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 1)
I20240503 07:59:31.925127 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.925503 14685 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:31.926614 6154 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.969439 14523 consensus_queue.cc:1035] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.972348 14564 consensus_queue.cc:1035] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.021101 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:32.026079 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.027531 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027663 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027820 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.027827 6408 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027897 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.027827 6670 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.028069 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.028069 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.030298 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.030392 14674 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.030512 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.030601 14690 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:32.031075 14691 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.031302 14674 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.031911 14691 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.032817 14689 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.033443 14690 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.033516 6540 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.033787 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.033804 14689 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.034794 14690 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:32.034880 14690 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.035214 14690 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.035302 14690 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.035579 14626 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.035332 14690 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.035755 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.035790 14690 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.035960 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.036167 14690 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.036453 14626 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.037189 14691 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.037258 14691 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:32.037552 14626 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.037617 14626 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:32.037593 14691 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.037667 14691 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.037686 14691 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.037786 14691 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.037998 14626 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.038090 14626 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.038120 14626 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.038125 14691 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.038251 14691 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.038221 14626 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.038478 14626 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.038637 14626 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.039077 14691 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.040225 14626 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.040812 14691 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.040905 14691 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.041287 14691 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.041384 14691 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.041416 14691 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.041517 14691 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.041790 14691 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.042078 14626 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.042145 14626 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:32.042568 14626 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.042659 14626 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.042687 14626 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.042783 14626 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
W20240503 07:59:32.042833 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:32.043026 14626 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.045980 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.046249 14658 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:32.047008 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.047273 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.047008 8822 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.049048 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.049228 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.049319 14685 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.049407 14699 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.050483 14699 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.053159 14699 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.053297 14699 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.054178 14699 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.054265 14699 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.054291 14699 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.054437 14699 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.054798 14699 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.055011 14699 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.056012 14699 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.056869 14699 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.056978 14699 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.057307 14699 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.057391 14699 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.057420 14699 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.057547 14699 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.057816 14699 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.059487 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 2)
I20240503 07:59:32.061543 14674 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.061628 14674 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.000s
I20240503 07:59:32.061658 14689 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.061750 14689 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 07:59:32.062009 14674 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062085 14674 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.062104 14674 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.062192 14674 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062209 14689 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062307 14689 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.062337 14689 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.062438 14674 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.062463 14689 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062727 14689 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.063661 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.063939 14657 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
W20240503 07:59:32.130388 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:32.133592 14564 raft_consensus.cc:482] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.133697 14564 raft_consensus.cc:504] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.134109 14564 leader_election.cc:288] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.134402 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.134512 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:32.134534 6560 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:32.134631 6430 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:32.135115 6625 leader_election.cc:302] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.135481 14564 raft_consensus.cc:2781] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.135543 14564 raft_consensus.cc:482] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.135582 14564 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.136545 14564 raft_consensus.cc:504] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.136904 14564 leader_election.cc:288] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.137315 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.137416 6560 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.138127 6560 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:32.138427 6625 leader_election.cc:302] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.138814 14564 raft_consensus.cc:2781] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.138902 14564 raft_consensus.cc:686] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:32.139024 14564 consensus_queue.cc:237] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.139601 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:32.139707 6430 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.140367 6430 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:32.141634 6154 catalog_manager.cc:5484] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.150163 14636 raft_consensus.cc:482] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.150254 14636 raft_consensus.cc:504] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.150655 14636 leader_election.cc:288] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.151057 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.151170 6560 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:32.151592 8776 leader_election.cc:302] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.151840 14636 raft_consensus.cc:2781] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.151903 14636 raft_consensus.cc:482] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.151953 14636 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.152871 14636 raft_consensus.cc:504] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.153460 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:32.153548 6429 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.153587 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.153673 6560 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.154486 6560 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:32.154505 6429 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:32.154735 8776 leader_election.cc:302] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.154882 14291 raft_consensus.cc:2781] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.153381 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:32.154946 14291 raft_consensus.cc:686] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:32.154983 6430 raft_consensus.cc:2362] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1. Re-sending same reply.
I20240503 07:59:32.155045 14291 consensus_queue.cc:237] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.156158 14636 leader_election.cc:288] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.157369 6156 catalog_manager.cc:5484] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.161342 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:32.167025 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.168272 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.169487 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:32.173662 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.174958 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 3)
I20240503 07:59:32.191150 14680 raft_consensus.cc:482] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.191252 14680 raft_consensus.cc:504] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.191669 14680 leader_election.cc:288] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.191927 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.192005 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:32.192118 8842 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.192127 8975 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.192409 8516 leader_election.cc:302] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.192636 14680 raft_consensus.cc:2781] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.192682 14680 raft_consensus.cc:482] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.192718 14680 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.193434 14680 raft_consensus.cc:504] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.193720 14680 leader_election.cc:288] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.194027 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:32.194029 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.194125 8975 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.194126 8842 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.194965 8842 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.194965 8975 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.195225 8516 leader_election.cc:302] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.195524 14680 raft_consensus.cc:2781] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.195605 14680 raft_consensus.cc:686] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:32.195724 14680 consensus_queue.cc:237] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.197853 6154 catalog_manager.cc:5484] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.278659 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.278791 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.280748 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.281301 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:32.281437 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:32.281436 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:32.281594 14706 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:32.281739 14708 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:32.281793 14706 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.281919 14707 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:32.281955 14706 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282055 14707 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.282063 14708 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:32.282166 14707 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282271 14708 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282681 14707 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.282660 14706 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.282817 14708 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.285202 14708 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.285224 14706 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.285288 14706 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.285305 14708 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.285686 14708 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:32.285781 14706 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:32.286343 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
I20240503 07:59:32.286509 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
I20240503 07:59:32.287096 14707 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.287170 14707 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.287520 14707 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:32.288209 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
W20240503 07:59:32.341648 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.382753 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:32.388303 8955 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388396 6670 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388574 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.388648 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.388689 6279 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388921 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.390411 14689 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.390852 14690 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:32.391393 14689 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.391723 14690 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.392071 14710 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.392791 14689 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.392890 14689 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.392918 14690 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:32.392982 14690 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:32.393321 14689 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393361 14690 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393419 14689 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.393448 14690 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.393448 14689 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.393476 14690 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.393573 14690 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393563 14689 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393822 14690 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.393834 14689 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.393896 14710 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.396108 14710 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.396203 14710 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.396711 14710 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.396840 14710 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.396920 14710 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.397048 14710 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.398068 14710 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:32.403946 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:32.421985 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.438863 14573 raft_consensus.cc:482] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.438963 14573 raft_consensus.cc:504] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.439309 14573 leader_election.cc:288] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:32.439553 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.439697 8975 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.439829 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:32.439942 6690 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.439965 6234 leader_election.cc:302] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:32.440121 14573 raft_consensus.cc:2781] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.440188 14573 raft_consensus.cc:482] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.440233 14573 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.441376 14573 raft_consensus.cc:504] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.441748 14573 leader_election.cc:288] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:32.444406 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.444519 8975 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.444625 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:32.444741 6690 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.445641 6690 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.446012 6234 leader_election.cc:302] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:32.446193 14573 raft_consensus.cc:2781] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.446261 14573 raft_consensus.cc:686] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:32.446264 8975 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.446393 14573 consensus_queue.cc:237] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
W20240503 07:59:32.447997 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.448885 6156 catalog_manager.cc:5484] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:32.491298 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.492318 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.541538 14291 consensus_queue.cc:1035] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.547068 14291 consensus_queue.cc:1035] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.555138 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:32.558930 6691 raft_consensus.cc:1261] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.558954 8976 raft_consensus.cc:1261] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.559289 14709 consensus_queue.cc:1035] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.559448 14573 consensus_queue.cc:1035] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.562260 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.562546 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.563038 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.563362 14685 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.563668 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.564891 14725 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:32.565802 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.566303 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.567458 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.567744 14724 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:32.569547 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41] alter complete (version 1)
I20240503 07:59:32.590694 14680 consensus_queue.cc:1035] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.593869 14727 consensus_queue.cc:1035] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:32.636518 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:32.643007 14523 consensus_queue.cc:1035] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.645790 14427 consensus_queue.cc:1035] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:32.647678 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.652133 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.670861 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.675283 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.675585 14685 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.676337 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.676853 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.677088 14725 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:32.677124 14724 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:32.682986 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41] alter complete (version 2)
W20240503 07:59:32.717115 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.725323 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:32.730978 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731298 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.731336 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731585 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.731632 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731853 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.733791 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.733872 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.734002 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.734133 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.734156 14732 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.734325 8559 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.734539 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.735273 14732 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.735976 6540 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.736184 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.736388 14734 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.736891 14710 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.736900 14731 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.737504 14734 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.737942 14731 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.738770 14734 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.738811 14710 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.738864 14734 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.739220 14731 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.739308 14731 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.739964 14710 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.739918 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.740042 14710 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.740001 8955 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.740170 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.740247 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.740428 14710 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.740533 14710 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.740562 14710 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.740664 14710 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.741205 14710 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.742345 14689 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.739284 14734 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.742553 14734 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.742588 14734 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.742702 14734 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.742873 14732 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.742949 14732 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:32.742949 14734 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 07:59:32.743093 14734 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.743201 14689 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.743558 14732 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.743661 14732 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.743687 14732 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.743790 14732 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.743978 14734 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.744051 14732 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.745770 14734 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.745921 14734 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.746390 14734 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.746516 14734 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.746577 14734 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.746706 14689 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.746734 14734 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.746790 14689 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.747118 14734 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.747193 14689 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.747289 14689 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.747282 14710 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.747315 14689 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.747422 14689 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.747668 14689 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.748214 14710 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:32.748682 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.749053 14731 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.749207 14710 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.749284 14710 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.749655 14710 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.749755 14710 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.749784 14710 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.749929 14710 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.750207 14710 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.750995 14731 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.751065 14731 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.751217 14731 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.752270 14731 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:32.752470 14731 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.753329 14731 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.754104 14738 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.754936 14738 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.755918 14731 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.756040 14731 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.756490 14731 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.756580 14731 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.756609 14731 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.756727 14731 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.756981 14731 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.757437 14738 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.757508 14738 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:32.758024 14738 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.758126 14738 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.758154 14738 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.758291 14738 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.758637 14738 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.771960 14727 raft_consensus.cc:482] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.772032 14727 raft_consensus.cc:504] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.772296 14727 leader_election.cc:288] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:32.772629 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.772629 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.772739 6559 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.772739 8976 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.773090 8516 leader_election.cc:302] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.773253 14727 raft_consensus.cc:2781] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.773296 14727 raft_consensus.cc:482] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.773331 14727 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.774111 14727 raft_consensus.cc:504] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.774398 14727 leader_election.cc:288] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:32.774647 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.774662 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.774744 6559 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.774756 8976 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.775506 6559 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.775668 8976 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.775759 8514 leader_election.cc:302] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.775938 14727 raft_consensus.cc:2781] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.776010 14727 raft_consensus.cc:686] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:32.776134 14727 consensus_queue.cc:237] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.778544 6150 catalog_manager.cc:5484] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.786098 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:32.790575 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.791496 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.791808 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.795285 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41] alter complete (version 3)
W20240503 07:59:32.836045 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.851310 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.864740 14723 raft_consensus.cc:482] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.864866 14723 raft_consensus.cc:504] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.865343 14723 leader_election.cc:288] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.865571 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:32.865628 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:32.865700 8581 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.865756 8843 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.865967 6232 leader_election.cc:302] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.866137 14723 raft_consensus.cc:2781] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.866196 14723 raft_consensus.cc:482] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.866240 14723 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.867004 14723 raft_consensus.cc:504] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.867292 14723 leader_election.cc:288] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.867568 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:32.867583 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:32.867671 8843 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.867669 8581 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.868392 8843 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.868428 8581 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.868651 6232 leader_election.cc:302] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.869009 14723 raft_consensus.cc:2781] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.869302 14723 raft_consensus.cc:686] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:32.869458 14723 consensus_queue.cc:237] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.871479 6156 catalog_manager.cc:5484] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:32.897997 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.898140 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.900192 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.900759 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:32.900832 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:32.900945 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:32.901304 14746 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:32.901319 14748 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:32.901458 14748 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.901490 14746 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:32.901585 14748 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.901672 14746 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.902112 14747 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:32.902200 14748 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.902246 14747 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.902200 14746 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.902398 14747 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.902890 14747 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.904539 14748 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.904551 14746 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.904611 14746 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.904640 14748 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.904918 14748 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:32.904969 14746 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:32.905370 14747 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.905479 14747 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.905623 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
I20240503 07:59:32.905781 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
I20240503 07:59:32.906715 14747 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:32.907570 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
W20240503 07:59:32.960942 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.963039 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:32.968410 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.995208 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:33.002578 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:33.008710 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.008895 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.008991 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.009078 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.011073 14734 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.011234 14356 raft_consensus.cc:482] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.011291 14750 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.011309 14356 raft_consensus.cc:504] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.011750 14356 leader_election.cc:288] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:33.011919 14734 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.012017 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.012207 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.012662 14750 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.013778 14734 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.013871 14734 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.014290 14734 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.014381 14734 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.014413 14734 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.014472 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:33.014523 14734 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.014586 6298 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:33.014798 14734 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.014847 6494 leader_election.cc:302] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.014987 14356 raft_consensus.cc:2781] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.015028 14356 raft_consensus.cc:482] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.015061 14356 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.015079 14731 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:33.015867 14731 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.015910 14356 raft_consensus.cc:504] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.016199 14356 leader_election.cc:288] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:33.016525 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:33.016619 6298 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.017535 6298 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:33.017779 6494 leader_election.cc:302] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.018090 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:33.018200 6429 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:33.019235 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:33.019326 6430 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.020154 6430 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:33.020345 14731 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:33.020429 14731 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:33.022009 14731 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.022102 14731 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.022130 14731 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.022240 14731 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.022531 14731 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.022696 14749 raft_consensus.cc:2781] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.022836 14749 raft_consensus.cc:686] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:33.022953 14749 consensus_queue.cc:237] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.026618 14750 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.026700 14750 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:33.026594 6156 catalog_manager.cc:5484] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:33.027122 14750 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.027216 14750 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.027244 14750 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.027364 14750 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.027623 14750 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:33.064486 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:33.152536 14759 consensus_queue.cc:1035] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.159993 14680 consensus_queue.cc:1035] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.184214 14760 raft_consensus.cc:482] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.184330 14760 raft_consensus.cc:504] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.184710 14760 leader_election.cc:288] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:33.184985 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:33.185055 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:33.185119 8581 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.185197 6559 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.185405 6625 leader_election.cc:302] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.185552 14760 raft_consensus.cc:2781] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.185612 14760 raft_consensus.cc:482] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.185721 14760 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.187115 14760 raft_consensus.cc:504] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.187393 14760 leader_election.cc:288] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:33.187638 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:33.187731 8581 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.187707 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:33.187822 6559 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.188450 8581 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.188508 6559 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.188688 6625 leader_election.cc:302] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.188828 14760 raft_consensus.cc:2781] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.188915 14760 raft_consensus.cc:686] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:33.189064 14760 consensus_queue.cc:237] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.191566 6154 catalog_manager.cc:5484] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:33.192631 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:33.300232 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.301235 14723 consensus_queue.cc:1035] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.304814 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:33.305006 14723 consensus_queue.cc:1035] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.309516 6560 raft_consensus.cc:1261] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.309609 8581 raft_consensus.cc:1261] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.309939 14760 consensus_queue.cc:1035] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.310201 14760 consensus_queue.cc:1035] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.312400 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.312819 14773 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:33.314157 6150 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.315445 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.315861 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.316139 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.316550 14775 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:33.318086 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 1)
I20240503 07:59:33.318779 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.331682 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.331971 14774 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.419572 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:33.424103 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.424448 14773 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:33.424979 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.425254 14775 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:33.426627 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.427524 14774 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.429687 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 2)
W20240503 07:59:33.478113 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:33.533394 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:33.538445 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.538642 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.541018 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.544425 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 3)
I20240503 07:59:33.562163 14356 consensus_queue.cc:1035] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.565003 14756 consensus_queue.cc:1035] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.596261 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:33.601528 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.601787 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.602320 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602581 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.602721 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602952 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.603914 14781 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.604058 6277 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602320 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.604252 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.604399 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.604636 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.604856 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.605139 14781 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.606770 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.606988 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.607242 14783 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:33.607182 8955 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.607431 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.608052 14783 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.608942 14786 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.610517 14782 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.610831 14786 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.612586 6670 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.612860 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.613337 14782 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.615271 14787 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:33.616855 14787 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.618016 14787 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:33.618104 14787 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.618549 14787 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.618654 14787 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.618736 14787 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.618871 14787 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.619202 14787 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.620363 14782 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.620507 14782 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:33.620931 14786 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.621039 14786 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:33.621043 14782 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621136 14782 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.621170 14782 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.621282 14782 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621515 14786 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621610 14786 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.621642 14786 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.621747 14786 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.622047 14786 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.621564 14782 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.624943 14782 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.625967 14783 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:33.626039 14783 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:33.626202 14782 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.626435 14783 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.626543 14783 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.626582 14783 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.626690 14783 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.626956 14783 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.627025 14781 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.627094 14781 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240503 07:59:33.627117 14783 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:33.627525 14781 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.627616 14781 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.627645 14781 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.627911 14783 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.631176 14781 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.631508 14781 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:33.631685 14781 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.633728 14781 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.636376 14781 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.637800 14781 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:33.638121 14782 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.638235 14782 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:33.638484 14781 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.638628 14781 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.638821 14781 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.638957 14781 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.639084 14782 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.639209 14782 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.639267 14782 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.639434 14781 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.639428 14782 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.639710 14782 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.639880 14782 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.640705 14782 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.641466 14783 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:33.641541 14783 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:33.641899 14783 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.641978 14783 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.642002 14783 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.642109 14783 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.642323 14783 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.645115 14782 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.645223 14782 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:33.645776 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.645814 14782 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.645943 14782 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.646003 14782 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.645978 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.646139 14782 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.646528 14782 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.648501 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.649104 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:33.649251 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:33.649318 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:33.649397 14795 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:33.649526 14795 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.649610 14797 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:33.649653 14795 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.649827 14797 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.650002 14797 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.650195 14795 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.650301 14796 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:33.650432 14796 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:33.650609 14796 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.650611 14797 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.651042 14796 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.652565 14795 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.652680 14795 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653074 14795 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:33.653394 14796 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.653604 14797 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.653631 14796 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653748 14797 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653786 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
I20240503 07:59:33.653914 14796 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:33.654175 14797 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:33.654520 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
I20240503 07:59:33.654877 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
W20240503 07:59:33.671602 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.713645 14723 raft_consensus.cc:482] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.713747 14723 raft_consensus.cc:504] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.714072 14723 leader_election.cc:288] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.714397 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:33.714421 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:33.714531 8843 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.714532 8581 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.714779 6232 leader_election.cc:302] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.714964 14723 raft_consensus.cc:2781] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.715030 14723 raft_consensus.cc:482] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.715056 14723 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.716013 14723 raft_consensus.cc:504] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.716327 14723 leader_election.cc:288] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.716629 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:33.716748 8843 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.716725 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:33.716852 8581 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.717715 8581 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.717710 8843 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.717988 6234 leader_election.cc:302] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.718199 14723 raft_consensus.cc:2781] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.718282 14723 raft_consensus.cc:686] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:33.718429 14723 consensus_queue.cc:237] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.720522 6155 catalog_manager.cc:5484] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:33.749773 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:33.754778 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.754784 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.755031 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.755121 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.756755 14800 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:33.756909 14786 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.757866 14786 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.757891 14800 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.758692 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.758870 14786 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.758905 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.758960 14786 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.759075 14800 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:33.759157 14800 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:33.759423 14786 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759526 14786 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.759565 14786 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.759554 14800 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759665 14800 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.759698 14800 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.759678 14786 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759935 14786 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.759802 14800 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.760607 14781 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.760746 14800 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:33.762642 14781 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.763707 14781 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.763803 14781 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.764201 14781 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.764281 14781 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.764307 14781 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.764412 14781 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.764675 14781 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:33.772929 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.791182 14776 raft_consensus.cc:482] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.791281 14776 raft_consensus.cc:504] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.791707 14776 leader_election.cc:288] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:33.792004 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:33.792032 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:33.792130 8976 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.792145 6296 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.792371 6625 leader_election.cc:302] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.792541 14776 raft_consensus.cc:2781] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.792615 14776 raft_consensus.cc:482] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.792665 14776 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.793779 14776 raft_consensus.cc:504] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.794169 14776 leader_election.cc:288] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:33.794430 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:33.794430 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:33.794540 8976 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.794540 6296 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.795428 8976 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.795428 6296 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.795676 6625 leader_election.cc:302] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.795856 14776 raft_consensus.cc:2781] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.795928 14776 raft_consensus.cc:686] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:33.796066 14776 consensus_queue.cc:237] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.798311 6155 catalog_manager.cc:5484] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:33.830708 14723 raft_consensus.cc:482] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.830801 14723 raft_consensus.cc:504] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.831140 14723 leader_election.cc:288] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.831455 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:33.831555 8843 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.831740 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:33.831777 6234 leader_election.cc:302] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.831857 6690 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.832032 14723 raft_consensus.cc:2781] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.832094 14723 raft_consensus.cc:482] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.832134 14723 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.832845 14723 raft_consensus.cc:504] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.833110 14723 leader_election.cc:288] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.833370 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:33.833462 6690 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.834103 6690 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.834318 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:33.834340 6234 leader_election.cc:302] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.834415 8843 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.835047 8843 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.835253 14723 raft_consensus.cc:2781] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.835337 14723 raft_consensus.cc:686] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:33.835445 14723 consensus_queue.cc:237] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.837549 6155 catalog_manager.cc:5484] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:33.977959 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:33.989336 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.011155 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:34.040060 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.061609 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.075883 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.136077 14759 raft_consensus.cc:482] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.136190 14759 raft_consensus.cc:504] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.136696 14759 leader_election.cc:288] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:34.136981 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:34.136987 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:34.137096 6690 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:34.137121 6559 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:34.137341 8516 leader_election.cc:302] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:34.137552 14759 raft_consensus.cc:2781] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.137614 14759 raft_consensus.cc:482] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.137667 14759 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.138671 14759 raft_consensus.cc:504] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.138967 14759 leader_election.cc:288] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:34.139346 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:34.139387 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:34.139442 6690 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.139511 6559 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.140200 6690 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:34.140470 8516 leader_election.cc:302] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:34.140514 6559 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:34.140628 14759 raft_consensus.cc:2781] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.140707 14759 raft_consensus.cc:686] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:34.140866 14759 consensus_queue.cc:237] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.143806 6155 catalog_manager.cc:5484] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:34.177676 14427 consensus_queue.cc:1035] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.180289 14776 consensus_queue.cc:1035] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.194424 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:34.208214 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.260418 14806 consensus_queue.cc:1035] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.262837 14723 consensus_queue.cc:1035] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.305518 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:34.309371 6690 raft_consensus.cc:1261] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.309455 6559 raft_consensus.cc:1261] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.309861 14759 consensus_queue.cc:1035] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.310128 14759 consensus_queue.cc:1035] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.313726 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.315527 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.317348 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.317577 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.317678 14821 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:34.317739 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.317982 14822 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:34.320755 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.321872 6155 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.325567 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 1)
I20240503 07:59:34.329152 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.329602 14823 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.353701 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:34.358903 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359182 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.359223 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359475 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.359596 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359825 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.361136 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.361204 14824 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:34.361362 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.361783 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.361990 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.362376 14824 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.363554 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.363662 14824 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:34.363729 14824 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.363790 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.364199 14824 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.364365 14824 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.364400 14824 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.364754 14824 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.365124 14824 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.365324 14828 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:34.365573 14825 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.366524 14828 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.366649 14825 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.366616 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.366864 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.367615 14828 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:34.367688 14828 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:34.368106 14828 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.368193 14828 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.368217 14828 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.368306 14828 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.368558 14828 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.368957 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.369129 14832 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:34.369177 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.369975 14832 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.370980 14832 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:34.371052 14832 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.371480 14832 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.371587 14832 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.371620 14832 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.371734 14832 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.372016 14832 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.372001 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.372162 14832 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:34.372233 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.372989 14832 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.374886 14829 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:34.375841 14829 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.376484 14826 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:34.376758 14824 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:34.377427 14826 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.377513 14824 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.378270 14829 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:34.378393 14829 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:34.378538 14826 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:34.378640 14826 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:34.378779 14829 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.378882 14829 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.378921 14829 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379045 14829 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379145 14832 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:34.379217 14832 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:34.379331 14829 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.379376 14824 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:34.379348 14826 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.379441 14824 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:34.379456 14826 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379485 14826 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379592 14826 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.379611 14832 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379695 14832 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379724 14832 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379823 14832 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379818 14824 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379946 14824 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379981 14824 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.380091 14832 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.380107 14824 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.380369 14824 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.380375 14825 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.380451 14825 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:34.380555 14826 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.380920 14825 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.381019 14825 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.381052 14825 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.381162 14825 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.381405 14825 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.381562 14825 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.382378 14825 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.382462 14778 raft_consensus.cc:482] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.382537 14778 raft_consensus.cc:504] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.382861 14778 leader_election.cc:288] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:34.383679 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:34.383746 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:34.383790 8843 raft_consensus.cc:2454] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:34.384070 6496 leader_election.cc:302] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:34.384250 14778 raft_consensus.cc:2781] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 07:59:34.384266 6497 leader_election.cc:341] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:34.384294 14778 raft_consensus.cc:482] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.385007 14778 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.386086 14778 raft_consensus.cc:504] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.386334 14776 raft_consensus.cc:482] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.386404 14778 leader_election.cc:288] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:34.386418 14776 raft_consensus.cc:504] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.386732 14776 leader_election.cc:288] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:34.386791 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
W20240503 07:59:34.387066 6497 leader_election.cc:341] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:34.387137 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:34.387234 8975 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:34.387296 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:34.387394 6429 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:34.387405 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:34.387492 8843 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.387715 14825 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.387791 14825 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:34.387768 6628 leader_election.cc:302] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:34.387955 14776 raft_consensus.cc:2781] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.387995 14776 raft_consensus.cc:482] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.388028 14776 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.388178 14825 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.388265 14825 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.388290 14825 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.388398 8843 raft_consensus.cc:2454] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:34.388394 14825 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.388649 14825 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.388859 6496 leader_election.cc:302] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters: d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:34.389009 14778 raft_consensus.cc:2781] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.389015 14776 raft_consensus.cc:504] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.389097 14778 raft_consensus.cc:686] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:34.389204 14778 consensus_queue.cc:237] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.389336 14776 leader_election.cc:288] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:34.389585 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:34.389675 8975 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.390412 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:34.390507 8975 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:34.390496 6429 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.390763 6627 leader_election.cc:302] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.390924 14776 raft_consensus.cc:2781] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.391012 14776 raft_consensus.cc:686] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:34.391120 14776 consensus_queue.cc:237] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.391345 6429 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:34.391604 6155 catalog_manager.cc:5484] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:34.392593 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.393870 6156 catalog_manager.cc:5484] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:34.403052 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.416043 14723 consensus_queue.cc:1035] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.418495 14723 consensus_queue.cc:1035] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.427963 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:34.432687 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.432996 14821 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:34.434166 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.434458 14823 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.435932 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.436183 14822 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:34.438868 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 2)
I20240503 07:59:34.541458 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:34.546453 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.547036 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.547369 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.552695 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 3)
I20240503 07:59:34.556813 14704 raft_consensus.cc:482] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.556911 14704 raft_consensus.cc:504] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.557301 14704 leader_election.cc:288] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:34.557590 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:34.557585 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:34.557703 8581 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:34.557703 6690 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:34.557943 8912 leader_election.cc:302] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.558112 14704 raft_consensus.cc:2781] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.558167 14704 raft_consensus.cc:482] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.558205 14704 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.559120 14704 raft_consensus.cc:504] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.559410 14704 leader_election.cc:288] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:34.559684 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:34.559695 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:34.559783 8581 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.559779 6690 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.560478 6690 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:34.560478 8581 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:34.560742 8912 leader_election.cc:302] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.560914 14704 raft_consensus.cc:2781] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.561019 14704 raft_consensus.cc:686] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:34.561141 14704 consensus_queue.cc:237] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.563315 6148 catalog_manager.cc:5484] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:34.581804 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.591657 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:34.613097 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:34.654304 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.654436 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.656417 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.656987 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:34.657060 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:34.657332 14846 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:34.657449 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:34.657471 14846 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:34.657660 14846 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.657763 14848 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:34.657924 14848 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.658046 14848 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.658185 14846 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.658898 14847 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:34.659094 14847 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.659233 14847 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.659775 14848 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.660676 14847 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.661324 14846 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.661394 14846 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.661609 14846 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:34.662153 14848 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.662225 14848 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.662360 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.662638 14848 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:34.662832 14847 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.662901 14847 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.663180 14847 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:34.663233 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.663705 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.758107 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:34.763290 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763290 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763538 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.763538 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.763554 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763792 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.765764 14829 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:34.765874 14825 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.766592 14828 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:34.766656 14829 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.766732 14825 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.767406 14828 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.767959 14829 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:34.768031 14829 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:34.768453 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.768441 14829 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.768556 14829 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.768590 14829 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.768692 14829 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.768977 14829 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.769004 14828 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:34.769068 14828 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.769472 14828 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.769569 14828 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.769595 14828 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.769697 14828 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.769945 14828 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.771327 14825 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.771425 14825 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:34.771842 14825 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.771947 14825 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.771981 14825 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.772078 14825 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.772351 14825 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:34.774981 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.790868 6429 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.792021 14356 consensus_queue.cc:1035] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.794730 14853 consensus_queue.cc:1035] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.884141 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.895361 14856 consensus_queue.cc:1035] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.899500 14856 consensus_queue.cc:1035] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.998251 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 07:59:34.999013 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:35.006618 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.042444 14753 raft_consensus.cc:482] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.042547 14753 raft_consensus.cc:504] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.042918 14753 leader_election.cc:288] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.043258 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.043381 6559 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.043365 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.043476 8581 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.043736 6363 leader_election.cc:302] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.044102 14686 raft_consensus.cc:2781] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.044160 14686 raft_consensus.cc:482] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.044193 14686 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.045279 14686 raft_consensus.cc:504] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.045567 14686 leader_election.cc:288] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.045797 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.045853 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.045941 8581 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.045904 6559 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.046887 6559 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.046898 8581 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.047281 6363 leader_election.cc:302] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.047432 14753 raft_consensus.cc:2781] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.047533 14753 raft_consensus.cc:686] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:35.047642 14753 consensus_queue.cc:237] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.050334 6148 catalog_manager.cc:5484] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:35.051632 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:35.092316 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.103140 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.109166 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109287 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109475 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.109424 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109572 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.109666 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.111629 14864 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.111634 14825 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.112449 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112557 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112600 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112684 14825 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.112715 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.112828 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.112833 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.113464 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.113708 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.114984 14828 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.115417 14825 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.115492 14825 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.115746 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.115916 14825 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.115983 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.116012 14825 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.116041 14828 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.116043 14825 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.116134 14868 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.116138 14825 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.116425 14825 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.117296 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.117422 14864 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.117518 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.118041 14865 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.118551 14828 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.118634 14828 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.119082 14828 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.119182 14828 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.119220 14828 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.119366 14828 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.119400 14865 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.119663 14828 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.119709 14825 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.120556 14825 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:35.121336 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.121696 14825 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.121692 14865 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.121781 14825 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:35.121789 14865 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.122180 14825 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.122287 14825 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.122236 14865 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.122321 14825 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.122346 14865 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.122378 14865 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.122433 14825 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.122548 14865 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.122740 14825 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.122908 14865 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.123087 14865 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.124264 14865 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.125497 14865 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.125615 14865 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.126070 14865 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.126196 14865 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.126253 14865 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.126402 14865 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.126506 14867 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.126709 14865 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.127655 14867 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.128716 14868 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.128751 14864 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.128870 14864 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:35.129305 14864 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.129411 14864 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.129446 14864 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.129559 14864 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.129858 14864 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.130313 14868 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.130347 14864 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.130393 14868 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:35.130805 14868 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.130906 14868 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.130939 14868 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
W20240503 07:59:35.130949 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.131047 14868 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.131228 14864 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.131305 14868 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.132308 14864 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.132398 14864 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:35.132666 14867 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.132742 14867 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:35.133131 14864 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133239 14864 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.133183 14867 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133267 14864 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.133275 14867 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.133306 14867 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.133407 14867 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133489 14864 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133673 14867 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.133849 14864 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.134558 14806 raft_consensus.cc:482] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.134630 14806 raft_consensus.cc:504] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.134949 14806 leader_election.cc:288] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:35.135279 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:35.135386 6690 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:35.135689 6234 leader_election.cc:302] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:35.136171 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:35.136189 14806 raft_consensus.cc:2781] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.136252 14806 raft_consensus.cc:482] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.136291 14806 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.136278 6429 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:35.137348 14806 raft_consensus.cc:504] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.137710 14806 leader_election.cc:288] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:35.138113 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:35.138211 6429 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.138505 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:35.138595 6690 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.139081 6429 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:35.139487 6690 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
W20240503 07:59:35.139497 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 07:59:35.139720 6235 leader_election.cc:302] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.139890 14806 raft_consensus.cc:2781] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.139984 14806 raft_consensus.cc:686] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:35.140098 14806 consensus_queue.cc:237] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.142850 6150 catalog_manager.cc:5484] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:35.149704 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 07:59:35.159790 14844 consensus_queue.cc:1035] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.163247 14844 consensus_queue.cc:1035] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.163314 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:35.167068 14686 raft_consensus.cc:482] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.167157 14686 raft_consensus.cc:504] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.167600 14686 leader_election.cc:288] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:35.168085 8581 raft_consensus.cc:1261] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.168511 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:35.168643 8842 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.168726 6559 raft_consensus.cc:1261] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.169099 14862 consensus_queue.cc:1035] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.169610 6365 leader_election.cc:302] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.169768 14862 raft_consensus.cc:2781] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.169809 14862 raft_consensus.cc:482] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.169857 14862 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.170912 14862 raft_consensus.cc:504] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.171250 14862 leader_election.cc:288] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:35.172029 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:35.172138 8842 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.173063 8842 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.173523 6365 leader_election.cc:302] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.173689 14753 consensus_queue.cc:1035] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.173960 14753 raft_consensus.cc:2781] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.174052 14753 raft_consensus.cc:686] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:35.174191 14753 consensus_queue.cc:237] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.174647 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
I20240503 07:59:35.176298 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.177543 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:35.177657 6298 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.178359 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:35.178632 6298 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.180114 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
W20240503 07:59:35.181529 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.182274 14886 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.185029 6148 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
I20240503 07:59:35.185050 6155 catalog_manager.cc:5484] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.186969 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.187258 14885 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:35.188357 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.188820 14887 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.188998 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 1)
W20240503 07:59:35.196712 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.197345 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 07:59:35.197942 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.216382 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.229275 14859 raft_consensus.cc:482] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.229383 14859 raft_consensus.cc:504] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.229784 14859 leader_election.cc:288] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.230086 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.230207 6559 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.230279 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.230424 8975 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.230509 6625 leader_election.cc:302] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.230710 14859 raft_consensus.cc:2781] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.230758 14859 raft_consensus.cc:482] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.230799 14859 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.231601 14859 raft_consensus.cc:504] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.231956 14859 leader_election.cc:288] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.232163 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.232296 8975 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.233300 8975 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.233486 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.233580 6560 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.234266 6560 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.234464 6627 leader_election.cc:302] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:35.234942 14859 raft_consensus.cc:2781] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.235054 14859 raft_consensus.cc:686] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:35.235170 14859 consensus_queue.cc:237] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.237179 6150 catalog_manager.cc:5484] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.290171 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 07:59:35.293486 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 07:59:35.295421 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.295735 14885 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:35.295871 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.296124 14887 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.296979 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.297252 14886 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.301381 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 2)
W20240503 07:59:35.315912 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 07:59:35.341409 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:35.403426 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:35.408362 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.408746 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.408998 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.414213 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 3)
W20240503 07:59:35.433310 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:35.516965 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.517127 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.519039 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.519518 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:35.519631 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:35.519768 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:35.519853 14889 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:35.519891 14890 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:35.519987 14889 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:35.520062 14891 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:35.520073 14890 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.520175 14889 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520187 14890 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520224 14891 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.520352 14891 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520634 14889 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.520890 14891 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.520939 14890 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.522506 14889 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.522603 14889 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.522883 14889 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:35.522953 14891 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.523056 14891 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.523077 14890 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.523139 14890 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.523363 14891 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:35.523407 14890 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:35.523515 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
I20240503 07:59:35.523962 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
I20240503 07:59:35.524381 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
W20240503 07:59:35.524906 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.527825 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.540796 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 07:59:35.542044 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:35.551419 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:35.607445 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.612282 14686 consensus_queue.cc:1035] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.620543 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:35.622504 14862 consensus_queue.cc:1035] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.630327 8955 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.630589 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.631083 8559 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.631311 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.633131 6540 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.633322 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.634749 14867 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.637252 14896 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.638119 14896 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.639716 14897 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.640121 14867 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.644320 14897 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.648283 14867 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.648370 14867 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:35.648403 14896 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.648471 14896 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:35.648826 14867 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.648885 14896 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.648938 14867 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.648967 14896 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.648967 14867 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.648995 14896 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.649098 14867 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.649405 14867 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.649523 14896 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.650015 14896 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.653101 14897 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.653226 14897 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240503 07:59:35.653784 14897 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.653949 14897 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.654026 14897 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.654183 14897 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.656622 14897 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:35.669754 14759 raft_consensus.cc:482] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.669828 14759 raft_consensus.cc:504] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.670138 14759 leader_election.cc:288] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.670419 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.670506 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.670536 8976 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:35.670604 6559 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:35.670842 8516 leader_election.cc:302] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.671028 14759 raft_consensus.cc:2781] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.671079 14759 raft_consensus.cc:482] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.671115 14759 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.672271 14759 raft_consensus.cc:504] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.672598 14759 leader_election.cc:288] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.672938 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.673035 6559 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.673045 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.673127 8976 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.673890 6559 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:35.673920 8976 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:35.674229 8516 leader_election.cc:302] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.674412 14759 raft_consensus.cc:2781] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.674446 14806 consensus_queue.cc:1035] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.674707 14759 raft_consensus.cc:686] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:35.674839 14759 consensus_queue.cc:237] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.677665 6150 catalog_manager.cc:5484] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:35.678550 14723 consensus_queue.cc:1035] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.722406 14856 consensus_queue.cc:1035] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.726897 14856 consensus_queue.cc:1035] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.752368 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.758546 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.758837 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.759260 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.759374 6672 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.759505 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.759601 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.760962 14897 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.761639 14896 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.761862 14897 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.762271 14909 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.762559 14896 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.763211 6277 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763460 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.763463 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763718 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766091 14909 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.766279 8955 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.766520 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766839 14897 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.766911 14897 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 07:59:35.767153 14909 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.767222 14909 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:35.767316 14897 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767436 14897 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.767467 14897 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.767585 14897 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767644 14909 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767720 14909 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.767748 14909 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.767853 14897 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:35.767858 14909 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.768105 14909 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.769163 14867 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.769935 14867 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.763463 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.771070 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.772097 14867 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.772176 14867 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.772528 14867 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.772611 14867 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.772641 14867 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.772748 14867 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.773042 14867 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.774139 6408 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.774398 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.776183 14916 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.777201 8559 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.777282 14916 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.777422 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.778316 14916 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.778393 14916 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.778889 14916 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.778987 14916 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.779018 14916 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.779129 14916 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.779398 14916 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.779719 14913 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.780576 14913 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.781215 14896 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.781311 14896 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240503 07:59:35.781692 14896 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.781783 14896 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.781811 14896 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.781908 14896 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.782164 14896 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.782274 14896 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.782878 14896 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.784507 14896 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.784579 14896 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:35.785468 14896 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.785553 14896 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.785575 14896 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.785681 14896 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.785952 14896 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:35.785964 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.786610 14912 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.787510 14912 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.788301 14913 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.788385 14913 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:35.788715 14913 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.788818 14913 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.788851 14912 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.788872 14913 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.788971 14913 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.789155 14912 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.789314 14913 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.789693 14912 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.789814 14912 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.789916 14912 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.790082 14913 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.790052 14912 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.790498 14912 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.790921 14913 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.792490 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:35.793534 14913 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.793597 14913 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:35.793973 14913 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.794137 14913 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.794211 14913 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.794373 14913 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.796753 14913 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:35.797502 8976 raft_consensus.cc:1261] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.797569 6559 raft_consensus.cc:1261] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.797962 14843 consensus_queue.cc:1035] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.798722 14843 consensus_queue.cc:1035] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.801429 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.801702 14887 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.802111 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.803936 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.804179 14886 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.804085 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.804697 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.804980 14924 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:35.807441 6150 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.808920 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 1)
I20240503 07:59:35.826041 14667 raft_consensus.cc:482] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.826125 14667 raft_consensus.cc:504] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.826467 14667 leader_election.cc:288] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.826735 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.826728 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.826849 8575 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:35.826855 6559 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:35.827100 8776 leader_election.cc:302] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.827260 14667 raft_consensus.cc:2781] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.827313 14667 raft_consensus.cc:482] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.827366 14667 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.828007 14667 raft_consensus.cc:504] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.828276 14667 leader_election.cc:288] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.828615 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.828614 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.828711 8575 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.828711 6559 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.829558 8575 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:35.829558 6559 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:35.829821 8776 leader_election.cc:302] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.830085 14667 raft_consensus.cc:2781] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.830171 14667 raft_consensus.cc:686] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:35.830310 14667 consensus_queue.cc:237] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.832265 6155 catalog_manager.cc:5484] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.874724 14427 raft_consensus.cc:482] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.874810 14427 raft_consensus.cc:504] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.875092 14427 leader_election.cc:288] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.875361 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.875399 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.875500 8575 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.875499 8976 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.875742 6625 leader_election.cc:302] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.875898 14427 raft_consensus.cc:2781] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.875959 14427 raft_consensus.cc:482] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.876003 14427 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.876866 14427 raft_consensus.cc:504] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.877147 14427 leader_election.cc:288] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.877434 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.877522 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.877560 8976 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.877600 8575 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.878216 8575 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.878330 8976 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.878479 6625 leader_election.cc:302] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.878667 14427 raft_consensus.cc:2781] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.878744 14427 raft_consensus.cc:686] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:35.878860 14427 consensus_queue.cc:237] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.880817 6155 catalog_manager.cc:5484] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.912032 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:35.917513 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.917948 14887 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.918627 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.918939 14924 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:35.918977 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.919255 14886 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.924317 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 2)
W20240503 07:59:36.024842 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.025696 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:36.035336 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.038213 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.038975 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.041217 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 3)
W20240503 07:59:36.075618 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.145476 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.145614 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.147480 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.147992 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:36.148352 14929 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:36.148334 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:36.148466 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:36.148540 14929 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.148648 14929 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.148761 14931 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:36.148943 14931 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.149202 14930 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:36.149363 14930 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.149439 14929 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.149576 14930 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.150146 14930 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.151319 14929 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.151381 14929 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.151618 14929 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:36.152299 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.152338 14930 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.152402 14930 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.152664 14930 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:36.149114 14931 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.153821 14931 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.154199 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.155555 14931 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.155614 14931 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.155839 14931 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:36.156383 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.220166 14667 consensus_queue.cc:1035] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:36.222419 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.223193 14905 consensus_queue.cc:1035] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.232893 14753 raft_consensus.cc:482] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.232981 14753 raft_consensus.cc:504] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.233266 14753 leader_election.cc:288] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:36.233541 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:36.233633 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:36.233664 8843 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.233740 6298 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.233922 6365 leader_election.cc:302] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.234076 14753 raft_consensus.cc:2781] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.234130 14753 raft_consensus.cc:482] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.234155 14753 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.234911 14753 raft_consensus.cc:504] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.235234 14753 leader_election.cc:288] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:36.235529 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:36.235632 6298 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.235606 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:36.235705 8843 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.236397 6298 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.236419 8843 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.236671 6363 leader_election.cc:302] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.236866 14753 raft_consensus.cc:2781] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.237010 14753 raft_consensus.cc:686] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:36.237136 14753 consensus_queue.cc:237] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.239583 6150 catalog_manager.cc:5484] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.248960 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:36.258143 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.258179 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.258460 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.258522 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.260622 14896 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.260717 14909 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:36.261536 14896 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.261677 14909 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.262662 14896 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.262698 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.262735 14896 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.262928 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.263311 14896 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.263414 14896 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.263443 14896 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.263537 14896 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.263792 14896 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.265224 14912 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:36.266201 14912 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.267060 14909 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:36.267158 14909 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.267550 14909 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.267666 14909 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.267704 14909 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.267869 14909 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.268165 14909 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.269520 14912 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:36.269611 14912 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:36.270082 14912 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.270186 14912 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.270221 14912 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.270334 14912 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.270643 14912 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.353006 14859 raft_consensus.cc:482] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.353101 14859 raft_consensus.cc:504] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.353415 14859 leader_election.cc:288] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:36.353696 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:36.353737 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:36.353816 8575 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:36.353839 6298 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:36.354089 6625 leader_election.cc:302] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:36.354256 14859 raft_consensus.cc:2781] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.354344 14859 raft_consensus.cc:482] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.354393 14859 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.355419 14859 raft_consensus.cc:504] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.355731 14859 leader_election.cc:288] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:36.356087 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:36.356206 8575 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.356209 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:36.356304 6298 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.357100 8575 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:36.357283 6298 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:36.357445 6625 leader_election.cc:302] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:36.357604 14859 raft_consensus.cc:2781] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.357708 14859 raft_consensus.cc:686] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:36.357998 14859 consensus_queue.cc:237] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.359988 6150 catalog_manager.cc:5484] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:36.407457 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:36.411500 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:36.419432 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.439360 14859 consensus_queue.cc:1035] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.444350 14859 consensus_queue.cc:1035] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.484067 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:36.488139 6298 raft_consensus.cc:1261] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.488488 8581 raft_consensus.cc:1261] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.488592 14859 consensus_queue.cc:1035] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.489195 14427 consensus_queue.cc:1035] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.492120 6148 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.492792 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.493794 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.494159 14950 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:36.494869 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.495170 14949 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:36.495290 6150 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.496480 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.496803 14887 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:36.497359 6150 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.500597 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.500749 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 1)
W20240503 07:59:36.520156 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:36.521711 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.604854 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:36.611068 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.611411 14950 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:36.610810 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.611667 14949 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:36.612879 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.613158 14887 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:36.618748 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 2)
W20240503 07:59:36.682842 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.701711 14753 consensus_queue.cc:1035] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.704981 14928 consensus_queue.cc:1035] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:36.710593 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.733937 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:36.738978 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.740032 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.743027 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.744103 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 3)
W20240503 07:59:36.834723 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.840832 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:36.845472 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.845588 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.847157 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.847494 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.847822 6408 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.848104 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.848529 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.848819 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.849880 14956 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:36.850195 14957 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:36.850307 8559 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.850631 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.851207 14957 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.851249 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.851483 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.851619 14956 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.852730 14961 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.852908 14956 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:36.852890 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.852979 14956 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.853106 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.853400 14956 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.853487 14956 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.853514 14956 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.853641 14956 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.853731 14961 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.853891 14956 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.854106 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.854374 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.854859 14961 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.854984 14961 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.854979 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.855154 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.855437 14961 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.855532 14961 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.855564 14961 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.855696 14961 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.856789 6541 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.856886 14963 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.857023 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.857412 14961 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.857579 14961 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.859055 14961 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.860003 14963 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.861421 14961 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.861467 14958 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:36.861513 14961 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:36.862277 14958 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.862304 14961 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.862481 14961 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.862560 14961 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.862747 14961 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.863250 14958 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:36.863366 14958 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.863687 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.863840 14958 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.864023 14958 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.864092 14958 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.864110 14961 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:36.864274 14958 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.864506 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:36.864969 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:36.864961 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:36.865097 14967 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:36.865240 14967 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.865293 14958 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.865483 14967 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.866384 14958 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:36.867481 14958 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.867830 14969 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:36.868029 14969 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.868289 14969 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.868412 14967 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.868598 14968 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:36.869030 14969 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.871671 14967 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.871673 14968 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.871755 14967 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.871806 14968 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.872119 14967 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:36.872381 14969 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.872865 14968 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.872874 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.873414 14958 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:36.873543 14958 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:36.874018 14958 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.874114 14958 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.874142 14958 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.874265 14958 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.874497 14969 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.874531 14958 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.874795 14969 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:36.875413 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.875792 14968 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.875859 14968 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.876065 14968 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:36.876691 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.877244 14957 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:36.877350 14957 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 07:59:36.877511 14963 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.877606 14963 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:36.877861 14957 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.877971 14957 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.878005 14957 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.878139 14957 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.878430 14957 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.878561 14963 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.878666 14963 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.878698 14963 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.878813 14963 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.879088 14963 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:36.879259 14963 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.880139 14963 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.884250 14963 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.884323 14963 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:36.884655 14963 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.884739 14963 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.884794 14963 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.884939 14963 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.885262 14963 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.885424 14963 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.886288 14963 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.887418 14963 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.887513 14963 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.887925 14963 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.888043 14963 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.888083 14963 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.888196 14963 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.888478 14963 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240503 07:59:36.946274 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.959203 14928 raft_consensus.cc:482] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.959296 14928 raft_consensus.cc:504] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.959614 14928 leader_election.cc:288] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:36.959929 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:36.959952 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:36.960052 8581 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.960054 6560 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.960286 6363 leader_election.cc:302] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.960453 14928 raft_consensus.cc:2781] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.960506 14928 raft_consensus.cc:482] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.960530 14928 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.961421 14928 raft_consensus.cc:504] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.961701 14928 leader_election.cc:288] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:36.961988 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:36.962015 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:36.962091 6560 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.962100 8581 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.962814 8581 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.962941 6560 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.963131 6363 leader_election.cc:302] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.963279 14928 raft_consensus.cc:2781] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.963363 14928 raft_consensus.cc:686] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:36.963500 14928 consensus_queue.cc:237] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.965371 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:36.966090 6156 catalog_manager.cc:5484] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:36.970693 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:36.971555 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.971797 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.971778 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.972010 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.973922 14956 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:36.974504 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.974685 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.974815 14956 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.976218 14956 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:36.976302 14956 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.976466 14963 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
W20240503 07:59:36.976994 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:36.976749 14956 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.977221 14974 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:36.977244 14956 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.977280 14956 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.977355 14963 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.977387 14956 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.977664 14956 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.978129 14974 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.979707 14963 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.979779 14963 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.980525 14963 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.980612 14963 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.980638 14963 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.980746 14963 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.981029 14963 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.982335 14974 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:36.982446 14974 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:36.982875 14974 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.983018 14974 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.983068 14974 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.983251 14974 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.983577 14974 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:36.984396 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 07:59:36.985538 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd4400894d14f839670d209468f1929: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:37.011873 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.016990 14853 raft_consensus.cc:482] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.017083 14853 raft_consensus.cc:504] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.017464 14853 leader_election.cc:288] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:37.017741 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.017835 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:37.017871 8843 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:37.017962 8975 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:37.018126 6496 leader_election.cc:302] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.018291 14853 raft_consensus.cc:2781] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.018348 14853 raft_consensus.cc:482] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.018390 14853 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.019426 14853 raft_consensus.cc:504] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.019737 14853 leader_election.cc:288] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:37.020032 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:37.020073 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.020123 8975 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.020179 8843 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.020973 8843 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:37.021077 8975 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:37.021229 6496 leader_election.cc:302] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.021509 14853 raft_consensus.cc:2781] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.021601 14853 raft_consensus.cc:686] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:37.021734 14853 consensus_queue.cc:237] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.024148 6150 catalog_manager.cc:5484] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.035825 14844 raft_consensus.cc:482] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.035919 14844 raft_consensus.cc:504] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.036337 14844 leader_election.cc:288] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.036702 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.036792 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:37.036849 6560 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.036900 8581 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.037096 8910 leader_election.cc:302] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.037241 14844 raft_consensus.cc:2781] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.037300 14844 raft_consensus.cc:482] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.037335 14844 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.038950 14844 raft_consensus.cc:504] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.039278 14844 leader_election.cc:288] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.039502 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.039538 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:37.039609 6560 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.039616 8581 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.040495 8581 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.040495 6560 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.040812 8910 leader_election.cc:302] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.041066 14844 raft_consensus.cc:2781] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.041149 14844 raft_consensus.cc:686] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:37.041270 14844 consensus_queue.cc:237] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.043865 6150 catalog_manager.cc:5484] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:37.053076 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.109584 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:37.213280 14427 raft_consensus.cc:482] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.213380 14427 raft_consensus.cc:504] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.213701 14427 leader_election.cc:288] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:37.213946 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.213999 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.214077 8843 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:37.214135 6560 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:37.214388 6625 leader_election.cc:302] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.214561 14427 raft_consensus.cc:2781] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.214619 14427 raft_consensus.cc:482] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.214655 14427 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.215875 14427 raft_consensus.cc:504] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.216163 14427 leader_election.cc:288] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:37.216424 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.216477 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.216544 8843 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.216565 6560 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.217495 6560 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:37.217610 8843 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:37.217808 6625 leader_election.cc:302] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.218029 14427 raft_consensus.cc:2781] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.218147 14427 raft_consensus.cc:686] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:37.218291 14427 consensus_queue.cc:237] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.220551 6150 catalog_manager.cc:5484] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:37.282526 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:37.372349 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:37.376359 8842 raft_consensus.cc:1261] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.376720 14427 consensus_queue.cc:1035] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.376888 6560 raft_consensus.cc:1261] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.377306 14427 consensus_queue.cc:1035] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.380676 6150 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.381660 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.383313 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.383674 14991 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:37.385409 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.385527 6156 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.385903 14993 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:37.386121 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.389418 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.389775 14992 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.390570 14984 consensus_queue.cc:1035] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.391659 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 1)
W20240503 07:59:37.394142 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.396624 14928 consensus_queue.cc:1035] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:37.412698 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:37.470705 14844 consensus_queue.cc:1035] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.473464 14844 consensus_queue.cc:1035] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.474686 14853 consensus_queue.cc:1035] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.477552 14853 consensus_queue.cc:1035] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.494136 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:37.498935 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.499291 14991 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:37.499310 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.499575 14992 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.499946 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.500252 14993 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:37.506247 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 2)
W20240503 07:59:37.530040 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.548743 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.555933 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:37.559702 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.561957 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562120 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562260 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.562408 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.562652 6279 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562891 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.564646 15007 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:37.565236 15006 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.565918 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.565997 15007 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.565996 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.566159 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.566258 15006 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.566269 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.567631 15006 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.567723 15006 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.567981 15010 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:37.568192 15006 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.568301 15006 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.568336 15006 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.568461 15006 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.568795 15006 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:37.568625 15011 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:37.569139 15006 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.569202 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569244 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569451 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.569696 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569849 15011 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.569921 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.570029 15006 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.571517 15010 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.571861 15006 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.565189 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.571991 15006 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.572125 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.572423 15006 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.572503 15006 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.572508 15010 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:37.572535 15006 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.572582 15010 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:37.572916 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.573040 15014 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.572661 15006 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.573698 15010 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.573791 15010 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.573896 15010 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.573983 15006 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.574035 15010 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.574295 15010 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:37.575001 15011 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:37.575096 15011 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:37.575352 15014 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.575508 15011 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.575595 15011 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.575623 15011 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.575748 15011 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.576040 15011 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.579473 15007 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:37.579474 15014 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.579555 15014 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:37.579564 15007 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:37.579939 15014 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.580034 15014 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.580060 15014 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.580045 15007 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.580143 15007 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.580188 15007 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.580158 15014 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.580287 15006 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.580307 15007 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.580402 15014 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.580572 15007 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.581445 15014 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.581965 15006 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.582577 15014 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.583561 15006 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.583827 15006 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:37.584276 15006 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.584429 15006 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.584481 15006 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.584587 15006 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.584867 15006 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:37.587340 15014 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.587420 15014 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:37.587812 15014 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.587898 15014 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.587924 15014 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.588016 15014 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.588256 15014 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.588375 15014 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.589228 15014 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.590948 15014 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.591030 15014 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:37.591380 15014 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.591452 15014 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.591480 15014 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.591583 15014 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.591832 15014 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.607347 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:37.611970 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.612707 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.615487 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.617928 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 3)
W20240503 07:59:37.623849 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:37.624459 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.678319 14985 raft_consensus.cc:482] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.678444 14985 raft_consensus.cc:504] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.678822 14985 leader_election.cc:288] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:37.679131 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:37.679152 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:37.679261 6429 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:37.679286 6690 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:37.679581 6234 leader_election.cc:302] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:37.679730 14985 raft_consensus.cc:2781] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.679787 14985 raft_consensus.cc:482] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.679828 14985 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.680670 14985 raft_consensus.cc:504] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.681013 14985 leader_election.cc:288] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:37.681303 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:37.681326 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:37.681418 6429 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.681444 6690 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.682374 6429 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:37.682386 6690 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:37.682641 6235 leader_election.cc:302] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.682785 14985 raft_consensus.cc:2781] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.682860 14985 raft_consensus.cc:686] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:37.682989 14985 consensus_queue.cc:237] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.685518 6150 catalog_manager.cc:5484] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.722133 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.722296 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.724279 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.724751 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:37.724982 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:37.725040 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:37.725265 15022 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:37.725329 15021 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:37.725346 15023 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:37.725453 15022 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:37.725463 15023 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.725574 15023 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.725559 15021 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.725646 15022 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.725920 15021 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.726135 15023 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.726135 15022 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.726416 15021 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.728338 15023 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728340 15022 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728387 15021 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728412 15022 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728477 15023 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728489 15021 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728787 15021 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:37.728871 15023 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:37.728997 15022 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:37.729477 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.729523 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.729918 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.736845 14952 raft_consensus.cc:482] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.736934 14952 raft_consensus.cc:504] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.737277 14952 leader_election.cc:288] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:37.737533 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:37.737668 6296 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.737672 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.737782 8843 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.737996 6363 leader_election.cc:302] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.738148 14952 raft_consensus.cc:2781] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.738204 14952 raft_consensus.cc:482] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.738240 14952 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.739117 14952 raft_consensus.cc:504] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.739456 14952 leader_election.cc:288] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:37.739725 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:37.739743 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.739825 6296 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.739838 8842 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.740715 8842 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.740854 6296 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.741096 6363 leader_election.cc:302] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.741309 14952 raft_consensus.cc:2781] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.741395 14952 raft_consensus.cc:686] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:37.741521 14952 consensus_queue.cc:237] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.743726 6150 catalog_manager.cc:5484] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:37.767725 14984 raft_consensus.cc:482] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.767848 14984 raft_consensus.cc:504] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.768150 14984 leader_election.cc:288] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:37.768448 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:37.768473 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:37.768559 6296 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.768590 8976 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.768824 6363 leader_election.cc:302] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.768977 14984 raft_consensus.cc:2781] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.769034 14984 raft_consensus.cc:482] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.769078 14984 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.769915 14984 raft_consensus.cc:504] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.770202 14984 leader_election.cc:288] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:37.770418 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:37.770527 8976 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.770586 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:37.770689 6296 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.771250 8976 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.771368 6296 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.771613 6363 leader_election.cc:302] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.771754 14984 raft_consensus.cc:2781] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.771835 14984 raft_consensus.cc:686] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:37.771955 14984 consensus_queue.cc:237] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.774163 6150 catalog_manager.cc:5484] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:37.814859 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.826284 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:37.832142 8956 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.832398 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.834615 15011 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:37.834957 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.835458 15011 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.835199 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.835659 8560 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.835884 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.836481 15011 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:37.836565 15011 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:37.836983 15011 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.837074 15011 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.837102 15011 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.837204 15011 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.837447 15011 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:37.841234 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.841384 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:37.841729 15027 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:37.842762 15028 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:37.842917 15027 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.843640 15028 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.844048 15027 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:37.844147 15027 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.844637 15027 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.844744 15027 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.844797 15027 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.844938 15027 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.845265 15027 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 07:59:37.846403 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.846562 15028 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:37.846618 15028 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:37.847005 15028 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.847097 15028 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.847124 15028 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.847224 15028 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.847486 15028 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:37.875875 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.932976 14844 raft_consensus.cc:482] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.933081 14844 raft_consensus.cc:504] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.933524 14844 leader_election.cc:288] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.933872 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.934000 6559 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.933982 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:37.934093 8575 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.934295 8910 leader_election.cc:302] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.934597 14844 raft_consensus.cc:2781] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.934655 14844 raft_consensus.cc:482] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.934705 14844 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.935873 14844 raft_consensus.cc:504] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.936244 14844 leader_election.cc:288] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.936479 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.936594 6559 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.936712 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:37.936863 8575 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.937621 6559 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.937779 8575 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.937958 8910 leader_election.cc:302] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.938097 14844 raft_consensus.cc:2781] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.938184 14844 raft_consensus.cc:686] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:37.938303 14844 consensus_queue.cc:237] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.940865 6155 catalog_manager.cc:5484] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:38.020500 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.032732 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.044904 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:38.052227 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.061272 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.064613 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:38.068195 8575 raft_consensus.cc:1261] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.068303 6559 raft_consensus.cc:1261] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.068573 15000 consensus_queue.cc:1035] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.068717 14844 consensus_queue.cc:1035] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.072466 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.073763 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.074162 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.074455 14992 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:38.074589 6150 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.076316 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.076738 15041 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.077096 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.077782 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.078598 6148 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.076054 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.080302 15040 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.081465 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 1)
I20240503 07:59:38.093695 14940 consensus_queue.cc:1035] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.096850 14940 consensus_queue.cc:1035] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.123792 14928 consensus_queue.cc:1035] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.126530 14952 consensus_queue.cc:1035] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.187203 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 07:59:38.188987 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:38.193289 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.193423 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.193679 15041 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.193681 15040 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.193956 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.194221 14992 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
W20240503 07:59:38.198520 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:38.200034 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 2)
I20240503 07:59:38.302707 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:38.302819 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:38.308914 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.310362 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.310520 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.310623 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.313084 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.315804 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.315889 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.316049 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.316151 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.318042 15028 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:38.318833 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.318943 15028 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.318994 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.319094 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.319260 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.320297 6670 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.320508 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.323681 15050 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.323737 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323770 6541 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323932 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324024 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324251 15053 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:38.324554 8824 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.324761 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.325985 14984 consensus_queue.cc:1035] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.326215 15027 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:38.327090 15027 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.328081 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 3)
I20240503 07:59:38.329108 15052 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:38.329409 15050 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330011 15053 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330138 15052 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.331246 15049 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:38.332337 15050 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.332342 15028 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:38.332422 15028 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:38.332585 15049 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.332593 15027 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:38.332661 15027 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:38.332677 15053 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:38.332749 15053 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.332917 15028 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.333030 15028 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.333063 15028 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.333139 15027 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.333220 15050 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:38.333179 15028 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.333241 15027 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.333268 15027 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.333364 15027 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.333444 15028 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.333588 15027 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.334805 15050 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.334877 15050 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.334898 15050 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.335000 15050 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.335265 15050 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:38.335436 15050 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.336148 15050 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.336679 15053 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.336802 15053 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.336835 15053 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.336936 15053 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.337203 15053 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:38.338912 15053 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:38.339661 15052 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:38.339756 15052 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:38.340034 15053 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.340201 15052 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.340307 15052 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.340343 15052 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.340458 15052 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.340732 15052 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.340922 15053 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:38.340993 15053 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:38.341491 15053 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.341634 15053 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.341701 15053 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.341257 14928 consensus_queue.cc:1035] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.341883 15053 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.342235 15053 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.342334 15049 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:38.342433 15049 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 07:59:38.342864 15049 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.342972 15049 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.343016 15049 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.343163 15049 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.343492 15049 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.350200 15050 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.350288 15050 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:38.350850 15050 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.350962 15050 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.350997 15050 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.351109 15050 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.351358 15050 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.351548 15050 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.352948 15050 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.359978 15050 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.360137 15050 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.360498 15050 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.360590 15050 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.360620 15050 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.360764 15050 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.361054 15050 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:38.375485 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.428201 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.428323 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.430336 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.430883 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:38.430883 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:38.430963 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:38.431222 15066 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:38.431396 15066 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:38.431602 15066 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.432096 15066 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.433997 15066 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.434074 15066 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.434481 15066 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:38.435174 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.436924 15065 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:38.437047 15065 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.437152 15065 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.437376 15064 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:38.437511 15064 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.437630 15064 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.438071 15065 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.439730 15065 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.439801 15065 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.440054 15065 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:38.440620 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.440857 15064 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.442562 15064 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.442642 15064 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.442947 15064 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:38.443464 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.455526 15005 raft_consensus.cc:482] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.455636 15005 raft_consensus.cc:504] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.456001 15005 leader_election.cc:288] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.456339 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:38.456466 6429 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.456480 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:38.456602 8843 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.456888 6627 leader_election.cc:302] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:38.457044 15005 raft_consensus.cc:2781] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.457108 15005 raft_consensus.cc:482] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.457134 15005 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.457949 15005 raft_consensus.cc:504] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.458245 15005 leader_election.cc:288] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.458498 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:38.458518 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:38.458593 8843 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.458603 6429 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.459306 6429 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.459307 8843 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.459559 6628 leader_election.cc:302] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.459720 15005 raft_consensus.cc:2781] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.459798 15005 raft_consensus.cc:686] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.459951 15005 consensus_queue.cc:237] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.462224 6156 catalog_manager.cc:5484] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.474171 15005 raft_consensus.cc:482] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.474261 15005 raft_consensus.cc:504] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.474591 15005 leader_election.cc:288] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:38.474933 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:38.475024 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:38.475061 6298 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.475138 8575 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.475293 6625 leader_election.cc:302] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:38.475443 15005 raft_consensus.cc:2781] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.475497 15005 raft_consensus.cc:482] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.475530 15005 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.476256 15005 raft_consensus.cc:504] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.476505 15005 leader_election.cc:288] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:38.476785 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:38.476836 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:38.476907 6298 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.476943 8575 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.477825 8575 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.477825 6298 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.478091 6625 leader_election.cc:302] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:38.478222 15005 raft_consensus.cc:2781] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.478297 15005 raft_consensus.cc:686] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.478417 15005 consensus_queue.cc:237] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.480901 6156 catalog_manager.cc:5484] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.531988 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:38.537221 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.537262 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.537465 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.537473 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.539023 15028 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:38.539320 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.539549 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.539925 15028 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.540753 15069 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:38.540964 15028 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:38.541041 15028 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.541451 15028 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.541535 15028 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.541563 15028 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.541669 15028 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.541700 15069 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.541893 15052 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:38.541910 15028 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.542721 15052 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.543191 15069 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:38.543277 15069 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.543956 15069 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.544041 15069 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.544070 15069 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.544203 15069 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.544718 15052 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:38.544831 15052 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.545254 15052 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.545342 15052 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.545379 15052 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.545486 15069 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:38.545496 15052 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.545765 15052 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.557619 14427 raft_consensus.cc:482] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.557714 14427 raft_consensus.cc:504] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.558079 14427 leader_election.cc:288] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:38.558432 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:38.558554 8843 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.558725 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:38.558844 6559 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.559124 6625 leader_election.cc:302] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:38.559373 14427 raft_consensus.cc:2781] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.559435 14427 raft_consensus.cc:482] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.559464 14427 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.560062 15034 raft_consensus.cc:482] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.560132 15034 raft_consensus.cc:504] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.560431 15034 leader_election.cc:288] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.560407 14427 raft_consensus.cc:504] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.560724 14427 leader_election.cc:288] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:38.560731 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:38.560881 6429 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:38.560875 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:38.560967 8976 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:38.561094 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:38.561168 8517 leader_election.cc:302] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.561193 6559 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.561319 15034 raft_consensus.cc:2781] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.561362 15034 raft_consensus.cc:482] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.561390 15034 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.561949 6559 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.562172 15034 raft_consensus.cc:504] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.562201 6625 leader_election.cc:302] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:38.562568 14427 raft_consensus.cc:2781] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.562614 15034 leader_election.cc:288] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.562590 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:38.562690 8843 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.562736 14427 raft_consensus.cc:686] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.562858 14427 consensus_queue.cc:237] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.562922 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:38.563002 6429 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.563599 8843 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.563791 6429 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:38.564133 8517 leader_election.cc:302] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.564397 15034 raft_consensus.cc:2781] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.564492 15034 raft_consensus.cc:686] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:38.564611 15034 consensus_queue.cc:237] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.564667 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:38.564759 8976 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.565639 8976 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:38.565661 6150 catalog_manager.cc:5484] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:38.567929 6148 catalog_manager.cc:5484] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:38.640483 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.674605 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:38.678755 6429 raft_consensus.cc:1261] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.679095 15034 consensus_queue.cc:1035] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.679170 8976 raft_consensus.cc:1261] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.679497 15034 consensus_queue.cc:1035] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.682077 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.682380 15041 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.682611 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]). Expected version 1 got 0
I20240503 07:59:38.683173 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.683243 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.683444 15080 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:38.683521 15040 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.684135 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]). Expected version 1 got 0
I20240503 07:59:38.689302 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 1)
W20240503 07:59:38.755965 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.791808 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:38.797223 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.797554 15041 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.797706 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.798022 15040 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.801735 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.802012 15080 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:38.802950 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 2)
W20240503 07:59:38.888700 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:38.902137 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.906383 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:38.911391 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.911795 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.913933 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.916520 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 3)
W20240503 07:59:38.917145 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.944195 15073 consensus_queue.cc:1035] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.944716 15073 consensus_queue.cc:1035] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.979795 15073 consensus_queue.cc:1035] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.983799 15073 consensus_queue.cc:1035] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.984558 15073 consensus_queue.cc:1035] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.988646 14427 consensus_queue.cc:1035] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.018988 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.019140 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.021430 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.021982 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:39.022006 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:39.022002 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:39.022418 15093 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:39.022470 15095 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:39.022631 15094 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:39.022647 15095 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.022758 15094 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.022763 15095 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.023069 15093 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:39.023324 15093 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.023361 15095 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.023438 15094 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.024219 15094 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.024940 15093 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.026530 15095 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.026616 15095 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.026937 15095 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:39.027565 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
I20240503 07:59:39.029206 15094 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.029281 15094 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.029693 15093 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.029763 15093 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.029980 15093 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:39.030470 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
I20240503 07:59:39.030573 15094 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:39.031148 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
W20240503 07:59:39.036003 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.044960 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.080870 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.113070 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.119096 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.119383 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.119410 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.119712 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.120203 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.120458 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.121683 15098 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:39.122431 8560 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.122651 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.122651 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.122792 15098 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.122897 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.123910 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:39.123992 15098 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:39.124080 15098 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:39.124524 15098 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.124627 15098 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.124660 15098 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.124867 15098 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.125023 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.125274 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.125865 15098 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.127149 15099 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:39.127827 15101 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.127900 6278 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.128125 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.128636 6541 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.128990 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.129673 15097 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:39.130306 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.130518 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.130704 15099 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.130738 15097 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.132953 15098 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:39.132288 15097 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:39.133427 15097 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:39.133848 15098 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.133946 15097 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.134050 15097 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.134073 15097 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.134164 15097 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.133064 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.134533 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.134981 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.135206 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.135640 15097 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.135782 15097 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:39.136718 15097 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.138623 15101 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.138818 15106 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:39.139364 15102 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.139814 15099 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:39.140129 15099 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:39.140276 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.140393 15102 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.140465 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.140578 15099 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.140672 15099 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.140705 15099 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.140831 15099 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.141083 15106 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.141541 15101 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.141669 15101 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:39.142156 15101 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.142235 15099 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.142297 15101 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.142370 15101 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.142514 15101 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.142848 15101 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.142974 15097 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:39.143055 15097 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:39.143399 15097 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.143484 15097 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.143513 15097 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.143635 15097 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.143872 15097 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.143985 15101 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.144912 15101 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.146865 15098 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:39.146952 15098 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:39.147449 15098 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.147548 15098 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.147581 15098 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.147593 15106 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:39.147656 15106 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:39.147696 15098 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.147974 15098 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.148046 15106 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.148136 15106 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.148165 15106 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.148294 15106 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.150408 15106 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:39.151146 15101 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.151247 15101 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:39.151639 15101 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.151752 15101 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.151798 15101 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.151921 15101 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.152915 15101 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:39.153110 15101 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.154011 15101 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.161590 15102 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.161681 15102 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:39.162042 15102 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.162132 15102 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.162164 15102 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.162290 15102 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.162532 15102 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.162716 15102 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.164064 15102 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.169732 14940 raft_consensus.cc:482] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.169806 14940 raft_consensus.cc:504] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.170131 14940 leader_election.cc:288] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.170454 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:39.170471 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:39.170552 6559 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:39.170583 8581 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:39.170985 6232 leader_election.cc:302] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:39.171229 14940 raft_consensus.cc:2781] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.171290 14940 raft_consensus.cc:482] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.171368 14940 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.171770 15102 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.171849 15102 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:39.172252 15102 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.172369 15102 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.172403 15102 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.172528 15102 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.172650 14940 raft_consensus.cc:504] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.172848 15102 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.173015 15102 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.173041 14940 leader_election.cc:288] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.174439 15102 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.175134 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:39.175241 6559 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.175230 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:39.175336 8581 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.176095 8581 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:39.176103 6559 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:39.176510 6232 leader_election.cc:302] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:39.176970 14985 raft_consensus.cc:2781] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.177127 14985 raft_consensus.cc:686] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:39.177251 14985 consensus_queue.cc:237] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.178089 15101 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.178174 15101 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.001s
I20240503 07:59:39.178562 15101 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.178746 15101 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.178824 15101 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.179004 15101 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.180197 15101 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:39.183171 15102 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.183264 15102 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:39.183611 6150 catalog_manager.cc:5484] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.183697 15102 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.183791 15102 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.183833 15102 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.183948 15102 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.184214 15102 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.201299 14944 raft_consensus.cc:482] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.201419 14944 raft_consensus.cc:504] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.201783 14944 leader_election.cc:288] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:39.202091 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:39.202209 6298 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:39.202247 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:39.202368 8581 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:39.202559 8910 leader_election.cc:302] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:39.202714 14944 raft_consensus.cc:2781] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.202772 14944 raft_consensus.cc:482] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.202809 14944 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.203603 14944 raft_consensus.cc:504] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.203858 14944 leader_election.cc:288] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:39.204126 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:39.204155 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:39.204237 6298 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.204237 8581 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.205155 8581 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:39.205155 6298 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:39.205422 8910 leader_election.cc:302] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:39.205588 14944 raft_consensus.cc:2781] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.205672 14944 raft_consensus.cc:686] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:39.205785 14944 consensus_queue.cc:237] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.207787 6148 catalog_manager.cc:5484] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:39.241195 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.247099 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.286736 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:39.297787 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.300087 15004 raft_consensus.cc:482] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.300184 15004 raft_consensus.cc:504] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.300563 15004 leader_election.cc:288] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.300921 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:39.300945 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:39.301048 6559 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:39.301054 8976 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:39.301307 8778 leader_election.cc:302] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:39.301497 14955 raft_consensus.cc:2781] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.301551 14955 raft_consensus.cc:482] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.301589 14955 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.302649 14955 raft_consensus.cc:504] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.302932 14955 leader_election.cc:288] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.303194 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:39.303282 6559 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.303195 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:39.303355 8976 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.304177 6559 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:39.304467 8776 leader_election.cc:302] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:39.304482 8976 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:39.304618 14955 raft_consensus.cc:2781] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.304719 14955 raft_consensus.cc:686] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:39.304885 14955 consensus_queue.cc:237] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.307137 6148 catalog_manager.cc:5484] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:39.309943 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.315850 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.379700 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.415431 15035 raft_consensus.cc:482] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.415555 15035 raft_consensus.cc:504] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.415939 15035 leader_election.cc:288] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:39.416292 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:39.416283 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:39.416428 6429 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:39.416455 8976 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:39.416739 6496 leader_election.cc:302] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:39.417025 15035 raft_consensus.cc:2781] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.417136 15035 raft_consensus.cc:482] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.417237 15035 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.418284 15035 raft_consensus.cc:504] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.418665 15035 leader_election.cc:288] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:39.419080 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:39.419129 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:39.419188 8975 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.419227 6429 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.419957 6429 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:39.420090 8975 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:39.420257 6497 leader_election.cc:302] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:39.420461 15035 raft_consensus.cc:2781] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.420596 15035 raft_consensus.cc:686] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:39.420729 15035 consensus_queue.cc:237] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.423180 6148 catalog_manager.cc:5484] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.538694 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:39.543291 8975 raft_consensus.cc:1261] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.543279 6429 raft_consensus.cc:1261] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.543730 15035 consensus_queue.cc:1035] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.543968 15096 consensus_queue.cc:1035] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.546836 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.547971 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.548426 15119 mvcc.cc:205] Tried to move back new op lower bound from 7023506143406202880 to 7023506142908743680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506143406202880}]
I20240503 07:59:39.548573 15122 mvcc.cc:205] Tried to move back new op lower bound from 7023506143406202880 to 7023506142908743680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506143406202880}]
I20240503 07:59:39.549748 15124 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.549880 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.550029 15125 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.550220 15124 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:39.550570 15126 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.550966 15126 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:39.552459 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.554541 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3] alter complete (version 1)
I20240503 07:59:39.554973 15125 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:39.585461 15076 consensus_queue.cc:1035] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.588193 14940 consensus_queue.cc:1035] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.603827 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6217
I20240503 07:59:39.608659 14944 consensus_queue.cc:1035] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:39.613623 8910 connection.cc:535] client connection to 127.0.99.129:37267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:39.614406 8910 consensus_peers.cc:485] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.614866 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6348
I20240503 07:59:39.626412 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6479
W20240503 07:59:39.626700 13845 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:39.625819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.130:41651 (local address 127.0.99.135:60133)
0503 07:59:39.625963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:39.626005 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":74}
W20240503 07:59:39.627806 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.635089 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6610
I20240503 07:59:39.642609 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8499
W20240503 07:59:39.649338 8776 connection.cc:535] client connection to 127.0.99.133:39949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:39.649621 8779 consensus_peers.cc:485] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:39.650115 15132 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:39.649758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:39949 (local address 127.0.99.135:41463)
0503 07:59:39.649853 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:39.649864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 07:59:39.649956 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 07:59:39.650032 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:39949: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
W20240503 07:59:39.650213 8776 consensus_peers.cc:485] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:39949: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.650342 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8761
W20240503 07:59:39.657717 8912 consensus_peers.cc:485] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.132:39733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.658046 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8894
I20240503 07:59:39.664227 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6126
2024-05-03T07:59:39Z chronyd exiting
I20240503 07:59:39.680066 398 test_util.cc:175] -----------------------------------------------
I20240503 07:59:39.680104 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0
LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx8_rf4_t8_l_3_1_1_1: Unrecognized error type. Please see the error log for more information.
Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 07:58:32.028163 398 test_util.cc:268] Using random seed: 368773108
I20240503 07:58:32.034085 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:32.034173 398 ts_itest-base.cc:113] --------------
I20240503 07:58:32.034204 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:58:32.034222 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:32.034236 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:32Z Disabled control of system clock
I20240503 07:58:32.042039 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36635
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36635
--enable_range_replica_placement=false with env {}
W20240503 07:58:32.139959 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.140143 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.140182 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.143095 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:32.143143 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.143163 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:32.143199 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:32.146678 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36635
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.147392 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.148387 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.155274 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.155443 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.156095 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.156894 410 server_base.cc:1034] running on GCE node
I20240503 07:58:32.157302 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.158103 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.159242 410 hybrid_clock.cc:648] HybridClock initialized: now 1714723112159213 us; error 51 us; skew 500 ppm
I20240503 07:58:32.163892 410 webserver.cc:460] Webserver started at http://127.0.99.190:35751/ using document root <none> and password file <none>
I20240503 07:58:32.164366 410 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.164427 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.164593 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:32.166138 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.166568 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.169514 410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 07:58:32.171396 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.172277 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:32.172374 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.172461 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.199031 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.199656 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.199812 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.205642 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36635
I20240503 07:58:32.205648 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36635 every 8 connection(s)
I20240503 07:58:32.206456 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:32.208709 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:32.213419 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: Bootstrap starting.
I20240503 07:58:32.215518 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:32.216192 477 log.cc:824] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:32.216580 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 410
I20240503 07:58:32.216696 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:32.217787 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: No bootstrap required, opened a new log
I20240503 07:58:32.220667 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } }
I20240503 07:58:32.220862 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:32.220925 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9224fc8cc494f44ab3b5bcceb6ad18b, State: Initialized, Role: FOLLOWER
I20240503 07:58:32.221276 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } }
I20240503 07:58:32.221383 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:32.221436 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:32.221524 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:32.222149 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } }
I20240503 07:58:32.222461 477 leader_election.cc:302] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9224fc8cc494f44ab3b5bcceb6ad18b; no voters:
I20240503 07:58:32.222731 477 leader_election.cc:288] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:32.222803 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:32.222986 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [term 1 LEADER]: Becoming Leader. State: Replica: b9224fc8cc494f44ab3b5bcceb6ad18b, State: Running, Role: LEADER
I20240503 07:58:32.223287 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } }
I20240503 07:58:32.223682 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:32.224951 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9224fc8cc494f44ab3b5bcceb6ad18b. Latest consensus state: current_term: 1 leader_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } } }
I20240503 07:58:32.225149 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:32.225386 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9224fc8cc494f44ab3b5bcceb6ad18b" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36635 } } }
I20240503 07:58:32.225484 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b [sys.catalog]: This master's current role is: LEADER
W20240503 07:58:32.228286 498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:58:32.228372 498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:58:32.228440 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:32.229264 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:32.234011 494 catalog_manager.cc:1347] Generated new cluster ID: 87177cb6d9674851848549d3e08b9a2c
I20240503 07:58:32.234072 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:32.247282 494 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:32.247972 494 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:32.261176 494 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b9224fc8cc494f44ab3b5bcceb6ad18b: Generated new TSK 0
I20240503 07:58:32.261668 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:32.271878 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 07:58:32.377797 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.378022 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.378077 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.381067 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.381175 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:32.384546 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.385314 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.386302 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.393765 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.393858 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.394665 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.395589 501 server_base.cc:1034] running on GCE node
I20240503 07:58:32.395915 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.396370 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.397538 501 hybrid_clock.cc:648] HybridClock initialized: now 1714723112397521 us; error 30 us; skew 500 ppm
I20240503 07:58:32.399111 501 webserver.cc:460] Webserver started at http://127.0.99.129:45035/ using document root <none> and password file <none>
I20240503 07:58:32.399597 501 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.399647 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.399806 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:32.401309 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "16f931607ac349629ac1e5eaa1d5d0a7"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.401739 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "16f931607ac349629ac1e5eaa1d5d0a7"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.404577 501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:32.406479 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.407351 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:32.407465 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "16f931607ac349629ac1e5eaa1d5d0a7"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.407534 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.430675 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.431243 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.431386 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.431850 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.432669 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.432714 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.432760 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.432816 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.440500 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:41375
I20240503 07:58:32.440534 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:41375 every 8 connection(s)
I20240503 07:58:32.441372 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:32.447209 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 501
I20240503 07:58:32.447311 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:32.449034 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36635
I20240503 07:58:32.449210 629 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.449373 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:32.449671 629 heartbeater.cc:511] Master 127.0.99.190:36635 requested a full tablet report, sending...
I20240503 07:58:32.450980 440 ts_manager.cc:194] Registered new tserver with Master: 16f931607ac349629ac1e5eaa1d5d0a7 (127.0.99.129:41375)
I20240503 07:58:32.452167 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:56345
W20240503 07:58:32.549849 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.550024 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.550074 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.552963 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.553066 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:32.556397 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.557127 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.558079 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.564930 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.565533 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.566359 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.566969 632 server_base.cc:1034] running on GCE node
I20240503 07:58:32.567291 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.567761 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.568943 632 hybrid_clock.cc:648] HybridClock initialized: now 1714723112568922 us; error 24 us; skew 500 ppm
I20240503 07:58:32.570544 632 webserver.cc:460] Webserver started at http://127.0.99.130:37599/ using document root <none> and password file <none>
I20240503 07:58:32.571010 632 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.571060 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.571216 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:32.572628 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "350a4a9aab3e483db61eaa681174e23d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.573096 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "350a4a9aab3e483db61eaa681174e23d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.575994 632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:32.577926 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.578774 632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:32.578862 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "350a4a9aab3e483db61eaa681174e23d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.578945 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.605473 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.605959 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.606086 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.606523 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.607264 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.607309 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.607343 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.607372 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.614863 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:44971
I20240503 07:58:32.614917 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:44971 every 8 connection(s)
I20240503 07:58:32.616103 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:32.623643 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36635
I20240503 07:58:32.623831 760 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.623847 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 632
I20240503 07:58:32.623952 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:32.624289 760 heartbeater.cc:511] Master 127.0.99.190:36635 requested a full tablet report, sending...
I20240503 07:58:32.625264 441 ts_manager.cc:194] Registered new tserver with Master: 350a4a9aab3e483db61eaa681174e23d (127.0.99.130:44971)
I20240503 07:58:32.625988 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:32.626005 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:37923
W20240503 07:58:32.738484 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.738669 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.738718 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.741693 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.741827 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:32.745534 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.746237 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.747187 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.754699 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.754859 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.755537 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.756325 763 server_base.cc:1034] running on GCE node
I20240503 07:58:32.756604 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.757105 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.758260 763 hybrid_clock.cc:648] HybridClock initialized: now 1714723112758235 us; error 35 us; skew 500 ppm
I20240503 07:58:32.759833 763 webserver.cc:460] Webserver started at http://127.0.99.131:46073/ using document root <none> and password file <none>
I20240503 07:58:32.760293 763 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.760341 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.760515 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:32.762041 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "37cc3d8dc1b846a3ba59fcef522fb73b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.762517 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "37cc3d8dc1b846a3ba59fcef522fb73b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.765569 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:32.767496 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.768400 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:32.768487 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "37cc3d8dc1b846a3ba59fcef522fb73b"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.768566 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.783241 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.783717 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.783839 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.784256 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.785038 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.785097 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.785135 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.785163 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.793453 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41003
I20240503 07:58:32.793483 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41003 every 8 connection(s)
I20240503 07:58:32.794217 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:32.801028 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 763
I20240503 07:58:32.801115 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:32.802110 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36635
I20240503 07:58:32.802335 891 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.802804 891 heartbeater.cc:511] Master 127.0.99.190:36635 requested a full tablet report, sending...
I20240503 07:58:32.802932 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 07:58:32.803766 441 ts_manager.cc:194] Registered new tserver with Master: 37cc3d8dc1b846a3ba59fcef522fb73b (127.0.99.131:41003)
I20240503 07:58:32.804311 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:58845
W20240503 07:58:32.902843 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:32.903018 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:32.903064 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:32.906929 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:32.907195 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:32.911238 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:32.911988 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:32.913019 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:32.920020 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.920518 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:32.921288 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:32.921854 894 server_base.cc:1034] running on GCE node
I20240503 07:58:32.922160 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:32.922613 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:32.923776 894 hybrid_clock.cc:648] HybridClock initialized: now 1714723112923753 us; error 36 us; skew 500 ppm
I20240503 07:58:32.925616 894 webserver.cc:460] Webserver started at http://127.0.99.132:35609/ using document root <none> and password file <none>
I20240503 07:58:32.926088 894 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:32.926136 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:32.926271 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:32.927726 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.928118 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.930967 894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:32.932914 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.933833 894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:32.933940 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d"
format_stamp: "Formatted at 2024-05-03 07:58:32 on dist-test-slave-7tt0"
I20240503 07:58:32.934010 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:32.966502 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:32.967483 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:32.967614 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:32.968070 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:32.968858 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:32.968906 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.968945 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:32.968973 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:32.977934 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:34599
I20240503 07:58:32.977957 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:34599 every 8 connection(s)
I20240503 07:58:32.978778 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:32.986497 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36635
I20240503 07:58:32.986675 1022 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:32.987107 1022 heartbeater.cc:511] Master 127.0.99.190:36635 requested a full tablet report, sending...
I20240503 07:58:32.987715 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 894
I20240503 07:58:32.987803 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:32.988000 441 ts_manager.cc:194] Registered new tserver with Master: 0271b74df4eb4a80be3d7e7cc5d0c28d (127.0.99.132:34599)
I20240503 07:58:32.988576 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:35705
I20240503 07:58:32.989655 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--builtin_ntp_servers=127.0.99.148:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 07:58:33.091940 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:33.092123 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:33.092175 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:33.095131 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:33.095240 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:33.098668 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36635
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:33.099409 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:33.100389 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:33.108114 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:33.108819 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:33.108886 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:33.109563 1025 server_base.cc:1034] running on GCE node
I20240503 07:58:33.109851 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:33.110417 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:33.111564 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714723113111551 us; error 25 us; skew 500 ppm
I20240503 07:58:33.113179 1025 webserver.cc:460] Webserver started at http://127.0.99.133:40485/ using document root <none> and password file <none>
I20240503 07:58:33.113687 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:33.113739 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:33.113881 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:33.115332 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "64fec6149db34c509eb82a89312e3ca6"
format_stamp: "Formatted at 2024-05-03 07:58:33 on dist-test-slave-7tt0"
I20240503 07:58:33.115760 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "64fec6149db34c509eb82a89312e3ca6"
format_stamp: "Formatted at 2024-05-03 07:58:33 on dist-test-slave-7tt0"
I20240503 07:58:33.118652 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:33.120522 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.121439 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 07:58:33.121539 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "64fec6149db34c509eb82a89312e3ca6"
format_stamp: "Formatted at 2024-05-03 07:58:33 on dist-test-slave-7tt0"
I20240503 07:58:33.121608 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:33.138741 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:33.139216 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:33.139353 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:33.139782 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:33.140527 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:33.140570 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.140614 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:33.140643 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:33.149137 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:46703
I20240503 07:58:33.149166 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:46703 every 8 connection(s)
I20240503 07:58:33.149907 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:33.154482 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 1025
I20240503 07:58:33.154583 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:33.157999 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36635
I20240503 07:58:33.158179 1153 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:33.158610 1153 heartbeater.cc:511] Master 127.0.99.190:36635 requested a full tablet report, sending...
I20240503 07:58:33.159475 440 ts_manager.cc:194] Registered new tserver with Master: 64fec6149db34c509eb82a89312e3ca6 (127.0.99.133:46703)
I20240503 07:58:33.159963 440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:43211
I20240503 07:58:33.168704 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:33.179982 440 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54130:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:33.196326 1087 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d8ef1269f4634997419a9175835d6 (DEFAULT_TABLE table=TestTable [id=5804a366b69e413184dcfeac3e9bf14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.196300 956 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d8ef1269f4634997419a9175835d6 (DEFAULT_TABLE table=TestTable [id=5804a366b69e413184dcfeac3e9bf14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.197229 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d8ef1269f4634997419a9175835d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.197247 956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d8ef1269f4634997419a9175835d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.197052 694 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d8ef1269f4634997419a9175835d6 (DEFAULT_TABLE table=TestTable [id=5804a366b69e413184dcfeac3e9bf14f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:33.197734 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d8ef1269f4634997419a9175835d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:33.201725 1170 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Bootstrap starting.
I20240503 07:58:33.202689 1171 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Bootstrap starting.
I20240503 07:58:33.203370 1172 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Bootstrap starting.
I20240503 07:58:33.203951 1170 tablet_bootstrap.cc:654] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.204535 1171 tablet_bootstrap.cc:654] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.204671 1170 log.cc:824] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:33.205003 1172 tablet_bootstrap.cc:654] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:33.205273 1171 log.cc:824] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:33.206073 1172 log.cc:824] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:33.206261 1170 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: No bootstrap required, opened a new log
I20240503 07:58:33.206424 1170 ts_tablet_manager.cc:1367] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:33.206797 1171 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: No bootstrap required, opened a new log
I20240503 07:58:33.206941 1171 ts_tablet_manager.cc:1367] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 07:58:33.210019 1172 tablet_bootstrap.cc:492] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: No bootstrap required, opened a new log
I20240503 07:58:33.210157 1172 ts_tablet_manager.cc:1367] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 07:58:33.210541 1171 raft_consensus.cc:348] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.210742 1171 raft_consensus.cc:374] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.210863 1171 raft_consensus.cc:729] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fec6149db34c509eb82a89312e3ca6, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.210808 1170 raft_consensus.cc:348] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.210983 1170 raft_consensus.cc:374] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.211036 1170 raft_consensus.cc:729] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0271b74df4eb4a80be3d7e7cc5d0c28d, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.211319 1171 consensus_queue.cc:260] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.211359 1170 consensus_queue.cc:260] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.212152 1171 ts_tablet_manager.cc:1398] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240503 07:58:33.212152 1170 ts_tablet_manager.cc:1398] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 07:58:33.212492 1153 heartbeater.cc:503] Master 127.0.99.190:36635 was elected leader, sending a full tablet report...
I20240503 07:58:33.212493 1022 heartbeater.cc:503] Master 127.0.99.190:36635 was elected leader, sending a full tablet report...
I20240503 07:58:33.212604 1172 raft_consensus.cc:348] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.213358 1172 raft_consensus.cc:374] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:33.213465 1172 raft_consensus.cc:729] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350a4a9aab3e483db61eaa681174e23d, State: Initialized, Role: FOLLOWER
I20240503 07:58:33.213795 1172 consensus_queue.cc:260] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.214335 760 heartbeater.cc:503] Master 127.0.99.190:36635 was elected leader, sending a full tablet report...
I20240503 07:58:33.214516 1172 ts_tablet_manager.cc:1398] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20240503 07:58:33.229508 1023 tablet.cc:2369] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:33.367256 761 tablet.cc:2369] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:33.400676 1154 tablet.cc:2369] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:33.416996 1176 raft_consensus.cc:482] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:33.417156 1176 raft_consensus.cc:504] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.418167 1176 leader_election.cc:288] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 350a4a9aab3e483db61eaa681174e23d (127.0.99.130:44971), 64fec6149db34c509eb82a89312e3ca6 (127.0.99.133:46703)
I20240503 07:58:33.422434 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d8ef1269f4634997419a9175835d6" candidate_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350a4a9aab3e483db61eaa681174e23d" is_pre_election: true
I20240503 07:58:33.422631 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d8ef1269f4634997419a9175835d6" candidate_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fec6149db34c509eb82a89312e3ca6" is_pre_election: true
I20240503 07:58:33.422694 714 raft_consensus.cc:2454] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0271b74df4eb4a80be3d7e7cc5d0c28d in term 0.
I20240503 07:58:33.422879 1107 raft_consensus.cc:2454] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0271b74df4eb4a80be3d7e7cc5d0c28d in term 0.
I20240503 07:58:33.423077 912 leader_election.cc:302] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0271b74df4eb4a80be3d7e7cc5d0c28d, 350a4a9aab3e483db61eaa681174e23d; no voters:
I20240503 07:58:33.423297 1176 raft_consensus.cc:2781] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:33.423377 1176 raft_consensus.cc:482] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:33.423416 1176 raft_consensus.cc:3037] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.424252 1176 raft_consensus.cc:504] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.424628 1176 leader_election.cc:288] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [CANDIDATE]: Term 1 election: Requested vote from peers 350a4a9aab3e483db61eaa681174e23d (127.0.99.130:44971), 64fec6149db34c509eb82a89312e3ca6 (127.0.99.133:46703)
I20240503 07:58:33.424960 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d8ef1269f4634997419a9175835d6" candidate_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fec6149db34c509eb82a89312e3ca6"
I20240503 07:58:33.424991 714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d8ef1269f4634997419a9175835d6" candidate_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350a4a9aab3e483db61eaa681174e23d"
I20240503 07:58:33.425092 714 raft_consensus.cc:3037] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.425074 1107 raft_consensus.cc:3037] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:33.426074 1107 raft_consensus.cc:2454] T dc3d8ef1269f4634997419a9175835d6 P 64fec6149db34c509eb82a89312e3ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0271b74df4eb4a80be3d7e7cc5d0c28d in term 1.
I20240503 07:58:33.426074 714 raft_consensus.cc:2454] T dc3d8ef1269f4634997419a9175835d6 P 350a4a9aab3e483db61eaa681174e23d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0271b74df4eb4a80be3d7e7cc5d0c28d in term 1.
I20240503 07:58:33.426329 912 leader_election.cc:302] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0271b74df4eb4a80be3d7e7cc5d0c28d, 350a4a9aab3e483db61eaa681174e23d; no voters:
I20240503 07:58:33.426484 1176 raft_consensus.cc:2781] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:33.426658 1176 raft_consensus.cc:686] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [term 1 LEADER]: Becoming Leader. State: Replica: 0271b74df4eb4a80be3d7e7cc5d0c28d, State: Running, Role: LEADER
I20240503 07:58:33.426967 1176 consensus_queue.cc:237] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } }
I20240503 07:58:33.429661 440 catalog_manager.cc:5484] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0271b74df4eb4a80be3d7e7cc5d0c28d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0271b74df4eb4a80be3d7e7cc5d0c28d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 34599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:33.454422 629 heartbeater.cc:503] Master 127.0.99.190:36635 was elected leader, sending a full tablet report...
I20240503 07:58:33.480410 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:33.481829 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 16f931607ac349629ac1e5eaa1d5d0a7 to finish bootstrapping
I20240503 07:58:33.487128 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 350a4a9aab3e483db61eaa681174e23d to finish bootstrapping
I20240503 07:58:33.491382 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 37cc3d8dc1b846a3ba59fcef522fb73b to finish bootstrapping
I20240503 07:58:33.496198 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0271b74df4eb4a80be3d7e7cc5d0c28d to finish bootstrapping
I20240503 07:58:33.500407 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 64fec6149db34c509eb82a89312e3ca6 to finish bootstrapping
I20240503 07:58:33.671636 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.671871 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:33.673735 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.675153 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.675758 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.806010 891 heartbeater.cc:503] Master 127.0.99.190:36635 was elected leader, sending a full tablet report...
I20240503 07:58:33.811466 1176 consensus_queue.cc:1035] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "350a4a9aab3e483db61eaa681174e23d" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.817128 1176 consensus_queue.cc:1035] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fec6149db34c509eb82a89312e3ca6" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:33.886190 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.886905 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.886981 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:33.888075 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:33.897686 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.093731 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.094786 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.095045 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.101207 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.105773 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.179733 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.179771 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.181273 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.181965 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.198014 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.279539 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.281038 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.282155 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.282259 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.293179 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.449170 440 ts_manager.cc:292] Set tserver state for 16f931607ac349629ac1e5eaa1d5d0a7 to MAINTENANCE_MODE
I20240503 07:58:34.618274 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.619371 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.622083 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.626042 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.638655 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.705631 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.707489 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.707679 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.708642 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.714581 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.807876 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.808507 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.808744 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.811434 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.825022 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.887607 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:34.889817 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.890995 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.897730 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.898049 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:34.934664 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 632
I20240503 07:58:35.103684 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.103955 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:35.104543 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.111024 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 07:58:35.158175 912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.130:44971: connect: Connection refused (error 111)
W20240503 07:58:35.158967 912 consensus_peers.cc:485] T dc3d8ef1269f4634997419a9175835d6 P 0271b74df4eb4a80be3d7e7cc5d0c28d -> Peer 350a4a9aab3e483db61eaa681174e23d (127.0.99.130:44971): Couldn't send request to peer 350a4a9aab3e483db61eaa681174e23d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:44971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:35.172144 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.172168 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:35.172926 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.173693 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.257782 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.260891 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.262993 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.271174 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:35.332116 563 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.333354 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.335005 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:35.343750 956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:35.367944 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 501
I20240503 07:58:35.373000 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 763
I20240503 07:58:35.377633 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 894
I20240503 07:58:35.381922 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 1025
I20240503 07:58:35.386091 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 410
2024-05-03T07:58:35Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (3378 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (3378 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 07:58:35.405789 398 test_util.cc:268] Using random seed: 372150746
I20240503 07:58:35.406452 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:35.406488 398 ts_itest-base.cc:113] --------------
I20240503 07:58:35.406503 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 07:58:35.406517 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:35.406530 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:35Z Disabled control of system clock
I20240503 07:58:35.413219 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38749
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:38749
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:58:35.509572 1625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.509758 1625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:35.509795 1625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.512650 1625 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:35.512706 1625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:35.512725 1625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:35.512741 1625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:35.512760 1625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:35.516242 1625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:38749
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:35.516994 1625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:35.517954 1625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:35.524367 1631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.525179 1633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.525691 1630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:35.526214 1625 server_base.cc:1034] running on GCE node
I20240503 07:58:35.526651 1625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:35.527429 1625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:35.528578 1625 hybrid_clock.cc:648] HybridClock initialized: now 1714723115528570 us; error 25 us; skew 500 ppm
I20240503 07:58:35.530215 1625 webserver.cc:460] Webserver started at http://127.0.99.190:46717/ using document root <none> and password file <none>
I20240503 07:58:35.530702 1625 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:35.530756 1625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:35.530907 1625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:35.532371 1625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "499ae78dd61e48719ecd90e1ddf724ea"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.532827 1625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "499ae78dd61e48719ecd90e1ddf724ea"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.535738 1625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:35.537652 1639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.538542 1625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:35.538656 1625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "499ae78dd61e48719ecd90e1ddf724ea"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.538731 1625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:35.564381 1625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:35.564894 1625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:35.565011 1625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:35.570562 1625 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:38749
I20240503 07:58:35.570569 1691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:38749 every 8 connection(s)
I20240503 07:58:35.571403 1625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:35.573673 1692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:35.577039 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 1625
I20240503 07:58:35.577150 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:35.578393 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea: Bootstrap starting.
I20240503 07:58:35.580289 1692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:35.580927 1692 log.cc:824] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:35.582290 1692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea: No bootstrap required, opened a new log
I20240503 07:58:35.584733 1692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } }
I20240503 07:58:35.584932 1692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:35.584990 1692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499ae78dd61e48719ecd90e1ddf724ea, State: Initialized, Role: FOLLOWER
I20240503 07:58:35.585379 1692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } }
I20240503 07:58:35.585515 1692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:35.585584 1692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:35.585678 1692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:35.586297 1692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } }
I20240503 07:58:35.586572 1692 leader_election.cc:302] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 499ae78dd61e48719ecd90e1ddf724ea; no voters:
I20240503 07:58:35.586799 1692 leader_election.cc:288] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:35.586902 1697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:35.587121 1697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [term 1 LEADER]: Becoming Leader. State: Replica: 499ae78dd61e48719ecd90e1ddf724ea, State: Running, Role: LEADER
I20240503 07:58:35.587390 1697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } }
I20240503 07:58:35.587702 1692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:35.588819 1699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 499ae78dd61e48719ecd90e1ddf724ea. Latest consensus state: current_term: 1 leader_uuid: "499ae78dd61e48719ecd90e1ddf724ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } } }
I20240503 07:58:35.588840 1698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "499ae78dd61e48719ecd90e1ddf724ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499ae78dd61e48719ecd90e1ddf724ea" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 38749 } } }
I20240503 07:58:35.588970 1698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:35.588970 1699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:35.589839 1706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:35.592666 1706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:35.597247 1706 catalog_manager.cc:1347] Generated new cluster ID: 7da6f1aa72414bbcb970f632e893df0d
I20240503 07:58:35.597311 1706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:35.620412 1706 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:35.621305 1706 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:35.628492 1706 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 499ae78dd61e48719ecd90e1ddf724ea: Generated new TSK 0
I20240503 07:58:35.629145 1706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:35.641139 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:35.739929 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.740113 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:35.740165 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.743146 1716 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:35.743206 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:35.743304 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:35.746726 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:35.747459 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:35.748512 1716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:35.755695 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.756314 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.757048 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:35.757831 1716 server_base.cc:1034] running on GCE node
I20240503 07:58:35.758155 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:35.758661 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:35.759819 1716 hybrid_clock.cc:648] HybridClock initialized: now 1714723115759793 us; error 40 us; skew 500 ppm
I20240503 07:58:35.761737 1716 webserver.cc:460] Webserver started at http://127.0.99.129:40981/ using document root <none> and password file <none>
I20240503 07:58:35.762198 1716 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:35.762251 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:35.762411 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:35.763849 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "50cf4bd5ec0d485a9cd183deb2693290"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.764266 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "50cf4bd5ec0d485a9cd183deb2693290"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.767273 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20240503 07:58:35.769235 1730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.770141 1716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:35.770236 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "50cf4bd5ec0d485a9cd183deb2693290"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.770306 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:35.786374 1716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:35.786799 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:35.786911 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:35.787290 1716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:35.788024 1716 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:35.788076 1716 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.788125 1716 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:35.788147 1716 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.796020 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:45011
I20240503 07:58:35.796079 1843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:45011 every 8 connection(s)
I20240503 07:58:35.796840 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:35.804387 1844 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38749
I20240503 07:58:35.804572 1844 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:35.805058 1844 heartbeater.cc:511] Master 127.0.99.190:38749 requested a full tablet report, sending...
I20240503 07:58:35.806043 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 1716
I20240503 07:58:35.806135 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:35.806564 1656 ts_manager.cc:194] Registered new tserver with Master: 50cf4bd5ec0d485a9cd183deb2693290 (127.0.99.129:45011)
I20240503 07:58:35.807935 1656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:33815
I20240503 07:58:35.808319 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:35.907227 1847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:35.907424 1847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:35.907467 1847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:35.910338 1847 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:35.910396 1847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:35.910460 1847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:35.914348 1847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--never_fsync=true
--heap_profile_path=/tmp/kudu.1847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:35.915048 1847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:35.916016 1847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:35.923360 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.924149 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:35.924978 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:35.925611 1847 server_base.cc:1034] running on GCE node
I20240503 07:58:35.925971 1847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:35.926455 1847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:35.927588 1847 hybrid_clock.cc:648] HybridClock initialized: now 1714723115927580 us; error 28 us; skew 500 ppm
I20240503 07:58:35.929172 1847 webserver.cc:460] Webserver started at http://127.0.99.130:45141/ using document root <none> and password file <none>
I20240503 07:58:35.929646 1847 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:35.929700 1847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:35.929845 1847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:35.931281 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1ea739c0f59d46c6adda1a33be3f0452"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.931700 1847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1ea739c0f59d46c6adda1a33be3f0452"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.934657 1847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240503 07:58:35.936524 1861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.937419 1847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:35.937508 1847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1ea739c0f59d46c6adda1a33be3f0452"
format_stamp: "Formatted at 2024-05-03 07:58:35 on dist-test-slave-7tt0"
I20240503 07:58:35.937578 1847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:35.957505 1847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:35.957967 1847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:35.958109 1847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:35.958561 1847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:35.959338 1847 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:35.959388 1847 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.959439 1847 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:35.959468 1847 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:35.967721 1847 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39279
I20240503 07:58:35.967747 1974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39279 every 8 connection(s)
I20240503 07:58:35.968513 1847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:35.973122 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 1847
I20240503 07:58:35.973222 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:35.975230 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:35.976460 1975 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38749
I20240503 07:58:35.976680 1975 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:35.977226 1975 heartbeater.cc:511] Master 127.0.99.190:38749 requested a full tablet report, sending...
I20240503 07:58:35.978157 1656 ts_manager.cc:194] Registered new tserver with Master: 1ea739c0f59d46c6adda1a33be3f0452 (127.0.99.130:39279)
I20240503 07:58:35.978718 1656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55207
W20240503 07:58:36.073176 1978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.073351 1978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:36.073392 1978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.076285 1978 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.076340 1978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.076413 1978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:36.079807 1978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--never_fsync=true
--heap_profile_path=/tmp/kudu.1978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.080540 1978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.081571 1978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:36.089553 1984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.090216 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.090350 1983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.090981 1978 server_base.cc:1034] running on GCE node
I20240503 07:58:36.091328 1978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.091845 1978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.093003 1978 hybrid_clock.cc:648] HybridClock initialized: now 1714723116092984 us; error 30 us; skew 500 ppm
I20240503 07:58:36.094789 1978 webserver.cc:460] Webserver started at http://127.0.99.131:43781/ using document root <none> and password file <none>
I20240503 07:58:36.095273 1978 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.095325 1978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.095499 1978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:36.097052 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "54b93477e23d4cd1a19fb7b2f1874bbf"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.097491 1978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "54b93477e23d4cd1a19fb7b2f1874bbf"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.100528 1978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:36.102635 1992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.103574 1978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:36.103677 1978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "54b93477e23d4cd1a19fb7b2f1874bbf"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.103751 1978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.142331 1978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.142887 1978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.143024 1978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.143515 1978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.144325 1978 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.144371 1978 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.144539 1978 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.144587 1978 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.153625 1978 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40747
I20240503 07:58:36.153668 2105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40747 every 8 connection(s)
I20240503 07:58:36.154527 1978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:36.160537 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 1978
I20240503 07:58:36.160619 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:36.162492 2106 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38749
I20240503 07:58:36.162616 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:36.162673 2106 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.163105 2106 heartbeater.cc:511] Master 127.0.99.190:38749 requested a full tablet report, sending...
I20240503 07:58:36.164047 1656 ts_manager.cc:194] Registered new tserver with Master: 54b93477e23d4cd1a19fb7b2f1874bbf (127.0.99.131:40747)
I20240503 07:58:36.164577 1656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:55381
W20240503 07:58:36.261163 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.261330 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:36.261376 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.264247 2109 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.264302 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.264376 2109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:36.267781 2109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--never_fsync=true
--heap_profile_path=/tmp/kudu.2109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.268486 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.269543 2109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 07:58:36.277602 2109 server_base.cc:1034] running on GCE node
W20240503 07:58:36.277750 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.278456 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.279021 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.279327 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.279750 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.280826 2109 hybrid_clock.cc:648] HybridClock initialized: now 1714723116280816 us; error 24 us; skew 500 ppm
I20240503 07:58:36.282439 2109 webserver.cc:460] Webserver started at http://127.0.99.132:43377/ using document root <none> and password file <none>
I20240503 07:58:36.282927 2109 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.282975 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.283130 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:36.284587 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1a04c0b1d7b94886b730ac71b41db6dd"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.285040 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1a04c0b1d7b94886b730ac71b41db6dd"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.288035 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:36.289999 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.290911 2109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:36.290994 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1a04c0b1d7b94886b730ac71b41db6dd"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.291069 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.301129 2109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.301640 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.301776 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.302189 2109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.302917 2109 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.302958 2109 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.302991 2109 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.303011 2109 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.311450 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:37565
I20240503 07:58:36.311475 2236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:37565 every 8 connection(s)
I20240503 07:58:36.312258 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:36.317118 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2109
I20240503 07:58:36.317241 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:36.319298 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--builtin_ntp_servers=127.0.99.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:36.320139 2237 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38749
I20240503 07:58:36.320318 2237 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.320881 2237 heartbeater.cc:511] Master 127.0.99.190:38749 requested a full tablet report, sending...
I20240503 07:58:36.321810 1656 ts_manager.cc:194] Registered new tserver with Master: 1a04c0b1d7b94886b730ac71b41db6dd (127.0.99.132:37565)
I20240503 07:58:36.322470 1656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:46791
W20240503 07:58:36.417011 2240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:36.417200 2240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:36.417241 2240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:36.420114 2240 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:36.420167 2240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:36.420241 2240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:36.423641 2240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:38749
--never_fsync=true
--heap_profile_path=/tmp/kudu.2240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:36.424402 2240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:36.425446 2240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:36.433380 2245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.434010 2246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:36.434759 2248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:36.435518 2240 server_base.cc:1034] running on GCE node
I20240503 07:58:36.435832 2240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:36.436326 2240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:36.437474 2240 hybrid_clock.cc:648] HybridClock initialized: now 1714723116437466 us; error 26 us; skew 500 ppm
I20240503 07:58:36.439138 2240 webserver.cc:460] Webserver started at http://127.0.99.133:35741/ using document root <none> and password file <none>
I20240503 07:58:36.439644 2240 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:36.439700 2240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:36.439854 2240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:36.441495 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "74298a57edb94e319ddcd7ff37455479"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.441926 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "74298a57edb94e319ddcd7ff37455479"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.444958 2240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:36.446923 2254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.447799 2240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:36.447892 2240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "74298a57edb94e319ddcd7ff37455479"
format_stamp: "Formatted at 2024-05-03 07:58:36 on dist-test-slave-7tt0"
I20240503 07:58:36.447963 2240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:36.460664 2240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:36.461180 2240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:36.461310 2240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:36.461750 2240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:36.462496 2240 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:36.462548 2240 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.462595 2240 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:36.462620 2240 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:36.470155 2240 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34529
I20240503 07:58:36.470188 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34529 every 8 connection(s)
I20240503 07:58:36.470927 2240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:36.474298 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2240
I20240503 07:58:36.474438 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:36.478684 2368 heartbeater.cc:347] Connected to a master server at 127.0.99.190:38749
I20240503 07:58:36.478880 2368 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:36.479318 2368 heartbeater.cc:511] Master 127.0.99.190:38749 requested a full tablet report, sending...
I20240503 07:58:36.480123 1656 ts_manager.cc:194] Registered new tserver with Master: 74298a57edb94e319ddcd7ff37455479 (127.0.99.133:34529)
I20240503 07:58:36.480618 1656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:46911
I20240503 07:58:36.488144 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:36.496258 1656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37376:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:36.512578 2040 tablet_service.cc:1467] Processing CreateTablet for tablet 581a5abafd104f7785ef4d4f1a4b2b68 (DEFAULT_TABLE table=TestTable [id=c2a694979bff458c9ada32741d57405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:36.512758 2302 tablet_service.cc:1467] Processing CreateTablet for tablet 581a5abafd104f7785ef4d4f1a4b2b68 (DEFAULT_TABLE table=TestTable [id=c2a694979bff458c9ada32741d57405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:36.512758 1909 tablet_service.cc:1467] Processing CreateTablet for tablet 581a5abafd104f7785ef4d4f1a4b2b68 (DEFAULT_TABLE table=TestTable [id=c2a694979bff458c9ada32741d57405f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:36.513540 2040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581a5abafd104f7785ef4d4f1a4b2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:36.513597 2302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581a5abafd104f7785ef4d4f1a4b2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:36.513612 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 581a5abafd104f7785ef4d4f1a4b2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:36.518137 2385 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Bootstrap starting.
I20240503 07:58:36.519309 2386 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Bootstrap starting.
I20240503 07:58:36.520009 2385 tablet_bootstrap.cc:654] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:36.520603 2385 log.cc:824] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:36.520910 2386 tablet_bootstrap.cc:654] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:36.521068 2387 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Bootstrap starting.
I20240503 07:58:36.521683 2386 log.cc:824] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:36.522866 2387 tablet_bootstrap.cc:654] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:36.522970 2386 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: No bootstrap required, opened a new log
I20240503 07:58:36.523150 2386 ts_tablet_manager.cc:1367] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 07:58:36.523240 2385 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: No bootstrap required, opened a new log
I20240503 07:58:36.523401 2385 ts_tablet_manager.cc:1367] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:36.523561 2387 log.cc:824] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:36.524978 2387 tablet_bootstrap.cc:492] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: No bootstrap required, opened a new log
I20240503 07:58:36.525133 2387 ts_tablet_manager.cc:1367] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 07:58:36.526058 2386 raft_consensus.cc:348] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.526257 2386 raft_consensus.cc:374] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:36.526304 2386 raft_consensus.cc:729] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea739c0f59d46c6adda1a33be3f0452, State: Initialized, Role: FOLLOWER
I20240503 07:58:36.526717 2386 consensus_queue.cc:260] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.526825 2385 raft_consensus.cc:348] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.526999 2385 raft_consensus.cc:374] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:36.527050 2385 raft_consensus.cc:729] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b93477e23d4cd1a19fb7b2f1874bbf, State: Initialized, Role: FOLLOWER
I20240503 07:58:36.527424 2386 ts_tablet_manager.cc:1398] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:36.527472 2385 consensus_queue.cc:260] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.527742 1975 heartbeater.cc:503] Master 127.0.99.190:38749 was elected leader, sending a full tablet report...
I20240503 07:58:36.528162 2106 heartbeater.cc:503] Master 127.0.99.190:38749 was elected leader, sending a full tablet report...
I20240503 07:58:36.528213 2385 ts_tablet_manager.cc:1398] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:36.528455 2387 raft_consensus.cc:348] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.528627 2387 raft_consensus.cc:374] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:36.528666 2387 raft_consensus.cc:729] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74298a57edb94e319ddcd7ff37455479, State: Initialized, Role: FOLLOWER
I20240503 07:58:36.529191 2387 consensus_queue.cc:260] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.530021 2387 ts_tablet_manager.cc:1398] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240503 07:58:36.530087 2368 heartbeater.cc:503] Master 127.0.99.190:38749 was elected leader, sending a full tablet report...
I20240503 07:58:36.540863 2391 raft_consensus.cc:482] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:36.540987 2391 raft_consensus.cc:504] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.542016 2391 leader_election.cc:288] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54b93477e23d4cd1a19fb7b2f1874bbf (127.0.99.131:40747), 74298a57edb94e319ddcd7ff37455479 (127.0.99.133:34529)
I20240503 07:58:36.545293 2060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581a5abafd104f7785ef4d4f1a4b2b68" candidate_uuid: "1ea739c0f59d46c6adda1a33be3f0452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" is_pre_election: true
I20240503 07:58:36.545310 2322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581a5abafd104f7785ef4d4f1a4b2b68" candidate_uuid: "1ea739c0f59d46c6adda1a33be3f0452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74298a57edb94e319ddcd7ff37455479" is_pre_election: true
I20240503 07:58:36.545492 2060 raft_consensus.cc:2454] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea739c0f59d46c6adda1a33be3f0452 in term 0.
I20240503 07:58:36.545495 2322 raft_consensus.cc:2454] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea739c0f59d46c6adda1a33be3f0452 in term 0.
I20240503 07:58:36.545852 1864 leader_election.cc:302] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea739c0f59d46c6adda1a33be3f0452, 54b93477e23d4cd1a19fb7b2f1874bbf; no voters:
I20240503 07:58:36.546039 2391 raft_consensus.cc:2781] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:36.546111 2391 raft_consensus.cc:482] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:36.546156 2391 raft_consensus.cc:3037] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:36.617213 2391 raft_consensus.cc:504] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.617636 2391 leader_election.cc:288] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [CANDIDATE]: Term 1 election: Requested vote from peers 54b93477e23d4cd1a19fb7b2f1874bbf (127.0.99.131:40747), 74298a57edb94e319ddcd7ff37455479 (127.0.99.133:34529)
I20240503 07:58:36.618022 2060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581a5abafd104f7785ef4d4f1a4b2b68" candidate_uuid: "1ea739c0f59d46c6adda1a33be3f0452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf"
I20240503 07:58:36.618034 2322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "581a5abafd104f7785ef4d4f1a4b2b68" candidate_uuid: "1ea739c0f59d46c6adda1a33be3f0452" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74298a57edb94e319ddcd7ff37455479"
I20240503 07:58:36.618144 2060 raft_consensus.cc:3037] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:36.618166 2322 raft_consensus.cc:3037] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:36.618971 2060 raft_consensus.cc:2454] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea739c0f59d46c6adda1a33be3f0452 in term 1.
I20240503 07:58:36.619172 2322 raft_consensus.cc:2454] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea739c0f59d46c6adda1a33be3f0452 in term 1.
I20240503 07:58:36.619237 1864 leader_election.cc:302] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea739c0f59d46c6adda1a33be3f0452, 54b93477e23d4cd1a19fb7b2f1874bbf; no voters:
I20240503 07:58:36.619406 2391 raft_consensus.cc:2781] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:36.619604 2391 raft_consensus.cc:686] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea739c0f59d46c6adda1a33be3f0452, State: Running, Role: LEADER
I20240503 07:58:36.619930 2391 consensus_queue.cc:237] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } }
I20240503 07:58:36.622680 1655 catalog_manager.cc:5484] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ea739c0f59d46c6adda1a33be3f0452 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "1ea739c0f59d46c6adda1a33be3f0452" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea739c0f59d46c6adda1a33be3f0452" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:36.655550 2107 tablet.cc:2369] T 581a5abafd104f7785ef4d4f1a4b2b68 P 54b93477e23d4cd1a19fb7b2f1874bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:36.655601 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 07:58:36.656968 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 50cf4bd5ec0d485a9cd183deb2693290 to finish bootstrapping
I20240503 07:58:36.662787 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1ea739c0f59d46c6adda1a33be3f0452 to finish bootstrapping
I20240503 07:58:36.666996 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 54b93477e23d4cd1a19fb7b2f1874bbf to finish bootstrapping
I20240503 07:58:36.671013 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1a04c0b1d7b94886b730ac71b41db6dd to finish bootstrapping
I20240503 07:58:36.675889 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 74298a57edb94e319ddcd7ff37455479 to finish bootstrapping
I20240503 07:58:36.680002 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 1716
W20240503 07:58:36.719693 1976 tablet.cc:2369] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:36.721663 2369 tablet.cc:2369] T 581a5abafd104f7785ef4d4f1a4b2b68 P 74298a57edb94e319ddcd7ff37455479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:36.855196 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.855765 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:36.855790 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.857055 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.935989 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.936913 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:36.937312 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:36.938896 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.033187 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.034086 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:37.035224 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.035235 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.193264 2499 consensus_queue.cc:1035] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54b93477e23d4cd1a19fb7b2f1874bbf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:37.195693 2499 consensus_queue.cc:1035] T 581a5abafd104f7785ef4d4f1a4b2b68 P 1ea739c0f59d46c6adda1a33be3f0452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74298a57edb94e319ddcd7ff37455479" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:37.244228 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:37.244230 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.245580 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.246898 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.314252 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.315173 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.315333 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:37.324236 2237 heartbeater.cc:503] Master 127.0.99.190:38749 was elected leader, sending a full tablet report...
I20240503 07:58:37.333289 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.412178 2171 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.412623 2040 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.413450 1909 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:37.415665 2302 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:37.445115 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 1847
I20240503 07:58:37.451915 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 1978
I20240503 07:58:37.456552 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2109
I20240503 07:58:37.460662 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2240
I20240503 07:58:37.465013 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 1625
2024-05-03T07:58:37Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (2079 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2079 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 07:58:37.485765 398 test_util.cc:268] Using random seed: 374230722
I20240503 07:58:37.486490 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:37.486531 398 ts_itest-base.cc:113] --------------
I20240503 07:58:37.486543 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 07:58:37.486554 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 07:58:37.486564 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:37Z Disabled control of system clock
I20240503 07:58:37.493606 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34925
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34925
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 07:58:37.670306 2592 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 07:58:37.670536 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:37.670573 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:37.670634 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:37.673586 2592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:37.673652 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:37.673683 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:37.673714 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:37.673738 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:37.677296 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34925
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:37.678119 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:37.679167 2592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:37.685910 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:37.686502 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:37.687093 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:37.687587 2592 server_base.cc:1034] running on GCE node
I20240503 07:58:37.688030 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:37.688926 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:37.690078 2592 hybrid_clock.cc:648] HybridClock initialized: now 1714723117690060 us; error 33 us; skew 500 ppm
I20240503 07:58:37.691917 2592 webserver.cc:460] Webserver started at http://127.0.99.190:36125/ using document root <none> and password file <none>
I20240503 07:58:37.692438 2592 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:37.692493 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:37.692653 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:37.694250 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "152cf5a0fb8c45389266683debe2a67e"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.694692 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "152cf5a0fb8c45389266683debe2a67e"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.697793 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:37.699784 2606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:37.700707 2592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:37.700829 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "152cf5a0fb8c45389266683debe2a67e"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.700904 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:37.715575 2592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:37.716084 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:37.716217 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:37.722211 2592 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34925
I20240503 07:58:37.722218 2658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34925 every 8 connection(s)
I20240503 07:58:37.723031 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:37.725397 2659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:37.729084 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2592
I20240503 07:58:37.729192 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:37.730145 2659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: Bootstrap starting.
I20240503 07:58:37.732141 2659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:37.732754 2659 log.cc:824] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:37.734208 2659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: No bootstrap required, opened a new log
I20240503 07:58:37.736707 2659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } }
I20240503 07:58:37.736927 2659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:37.736982 2659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152cf5a0fb8c45389266683debe2a67e, State: Initialized, Role: FOLLOWER
I20240503 07:58:37.737398 2659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } }
I20240503 07:58:37.737527 2659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:37.737582 2659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:37.737684 2659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:37.749110 2659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } }
I20240503 07:58:37.749449 2659 leader_election.cc:302] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 152cf5a0fb8c45389266683debe2a67e; no voters:
I20240503 07:58:37.749698 2659 leader_election.cc:288] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:37.750079 2664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:37.750280 2664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [term 1 LEADER]: Becoming Leader. State: Replica: 152cf5a0fb8c45389266683debe2a67e, State: Running, Role: LEADER
I20240503 07:58:37.750617 2664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } }
I20240503 07:58:37.750691 2659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:37.752234 2666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152cf5a0fb8c45389266683debe2a67e. Latest consensus state: current_term: 1 leader_uuid: "152cf5a0fb8c45389266683debe2a67e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } } }
I20240503 07:58:37.752349 2666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:37.752630 2665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "152cf5a0fb8c45389266683debe2a67e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152cf5a0fb8c45389266683debe2a67e" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34925 } } }
I20240503 07:58:37.752983 2665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e [sys.catalog]: This master's current role is: LEADER
W20240503 07:58:37.755151 2679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 07:58:37.755213 2679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 07:58:37.755280 2678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:37.756047 2678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:37.760613 2678 catalog_manager.cc:1347] Generated new cluster ID: cfe94b9bde6245b28b3dcca51e996623
I20240503 07:58:37.760677 2678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:37.767269 2678 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:37.768071 2678 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:37.773070 2678 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 152cf5a0fb8c45389266683debe2a67e: Generated new TSK 0
I20240503 07:58:37.773648 2678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:37.783380 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:37.884434 2683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:37.884599 2683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:37.884639 2683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:37.887671 2683 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:37.887724 2683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:37.887790 2683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:37.891222 2683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.2683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:37.891950 2683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:37.893601 2683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:37.900823 2689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:37.901441 2691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:37.901963 2688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:37.902796 2683 server_base.cc:1034] running on GCE node
I20240503 07:58:37.903138 2683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:37.903699 2683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:37.904825 2683 hybrid_clock.cc:648] HybridClock initialized: now 1714723117904811 us; error 29 us; skew 500 ppm
I20240503 07:58:37.906692 2683 webserver.cc:460] Webserver started at http://127.0.99.129:39033/ using document root <none> and password file <none>
I20240503 07:58:37.907214 2683 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:37.907279 2683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:37.907486 2683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:37.909179 2683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9a340c22b2014883b6bf4e891be77b34"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.909636 2683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9a340c22b2014883b6bf4e891be77b34"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.912644 2683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:37.914670 2697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:37.915599 2683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:37.915701 2683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9a340c22b2014883b6bf4e891be77b34"
format_stamp: "Formatted at 2024-05-03 07:58:37 on dist-test-slave-7tt0"
I20240503 07:58:37.915776 2683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:37.936677 2683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:37.937276 2683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:37.937431 2683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:37.937927 2683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:37.938786 2683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:37.938843 2683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:37.938899 2683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:37.938930 2683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:37.947358 2683 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:41775
I20240503 07:58:37.947398 2810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:41775 every 8 connection(s)
I20240503 07:58:37.948204 2683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:37.956418 2811 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:37.956609 2811 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:37.957065 2811 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:37.958113 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2683
I20240503 07:58:37.958202 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:37.958346 2623 ts_manager.cc:194] Registered new tserver with Master: 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:37.959417 2623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:41289
I20240503 07:58:37.960371 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:38.056865 2814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.057032 2814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.057085 2814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.059962 2814 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.060016 2814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.060096 2814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:38.063480 2814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.2814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.064230 2814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.065250 2814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.072386 2820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.073015 2822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.073648 2819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.074182 2814 server_base.cc:1034] running on GCE node
I20240503 07:58:38.074499 2814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.074975 2814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.076117 2814 hybrid_clock.cc:648] HybridClock initialized: now 1714723118076112 us; error 30 us; skew 500 ppm
I20240503 07:58:38.077759 2814 webserver.cc:460] Webserver started at http://127.0.99.130:32791/ using document root <none> and password file <none>
I20240503 07:58:38.078271 2814 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.078326 2814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.078486 2814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.079996 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c66f48bb18804160915381eac766f6fe"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.080432 2814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c66f48bb18804160915381eac766f6fe"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.083529 2814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:38.085665 2828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.086596 2814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:38.086694 2814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c66f48bb18804160915381eac766f6fe"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.086778 2814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.122352 2814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.122985 2814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.123145 2814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.123659 2814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.124516 2814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.124567 2814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.124619 2814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.124646 2814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.132649 2814 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39115
I20240503 07:58:38.132683 2941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39115 every 8 connection(s)
I20240503 07:58:38.133538 2814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:38.134975 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2814
I20240503 07:58:38.135073 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:38.137349 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:38.142215 2942 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.142444 2942 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.143106 2942 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.144102 2623 ts_manager.cc:194] Registered new tserver with Master: c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:38.144616 2623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:51999
W20240503 07:58:38.241712 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.241892 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.241940 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.244959 2945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.245020 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.245134 2945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:38.248528 2945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.2945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.249297 2945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.250348 2945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.260192 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.260632 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.261332 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.262023 2945 server_base.cc:1034] running on GCE node
I20240503 07:58:38.262362 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.262815 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.263959 2945 hybrid_clock.cc:648] HybridClock initialized: now 1714723118263943 us; error 29 us; skew 500 ppm
I20240503 07:58:38.265631 2945 webserver.cc:460] Webserver started at http://127.0.99.131:35457/ using document root <none> and password file <none>
I20240503 07:58:38.266114 2945 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.266167 2945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.266311 2945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.267769 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5294114bec9e421a866b4f999d3bb6bf"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.268167 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5294114bec9e421a866b4f999d3bb6bf"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.271173 2945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:38.273205 2959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.274089 2945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:38.274195 2945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5294114bec9e421a866b4f999d3bb6bf"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.274271 2945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.286162 2945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.286669 2945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.286784 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.287189 2945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.287936 2945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.287981 2945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.288014 2945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.288034 2945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.295826 2945 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40707
I20240503 07:58:38.295858 3072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40707 every 8 connection(s)
I20240503 07:58:38.296633 2945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:38.302850 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 2945
I20240503 07:58:38.302944 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:38.304155 3073 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.304345 3073 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.304854 3073 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.304904 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:38.305765 2622 ts_manager.cc:194] Registered new tserver with Master: 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131:40707)
I20240503 07:58:38.306354 2622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:43681
W20240503 07:58:38.405166 3076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.405354 3076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.405406 3076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.408277 3076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.408342 3076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.408425 3076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:38.411852 3076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.412567 3076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.413550 3076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.421399 3081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.422066 3082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.422897 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.423166 3076 server_base.cc:1034] running on GCE node
I20240503 07:58:38.423561 3076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.424047 3076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.425185 3076 hybrid_clock.cc:648] HybridClock initialized: now 1714723118425179 us; error 28 us; skew 500 ppm
I20240503 07:58:38.426826 3076 webserver.cc:460] Webserver started at http://127.0.99.132:40035/ using document root <none> and password file <none>
I20240503 07:58:38.427294 3076 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.427347 3076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.427502 3076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.428967 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f9266aaeb8404667ae633cb22af9d372"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.429373 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f9266aaeb8404667ae633cb22af9d372"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.432366 3076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:58:38.434477 3090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.435601 3076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:38.435714 3076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f9266aaeb8404667ae633cb22af9d372"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.435802 3076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.453254 3076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.453876 3076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.454027 3076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.454560 3076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.455495 3076 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.455550 3076 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.455601 3076 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.455636 3076 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.463506 3076 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43805
I20240503 07:58:38.463562 3203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43805 every 8 connection(s)
I20240503 07:58:38.464311 3076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:38.469709 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3076
I20240503 07:58:38.469815 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:38.471998 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:38.472699 3204 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.472916 3204 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.473464 3204 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.474421 2622 ts_manager.cc:194] Registered new tserver with Master: f9266aaeb8404667ae633cb22af9d372 (127.0.99.132:43805)
I20240503 07:58:38.474982 2622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:35133
W20240503 07:58:38.574345 3207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.574517 3207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.574559 3207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.577471 3207 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.577525 3207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.577592 3207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:38.581471 3207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.582190 3207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.583154 3207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.590739 3213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.591214 3215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.591958 3212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.592686 3207 server_base.cc:1034] running on GCE node
I20240503 07:58:38.593050 3207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.593839 3207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.594988 3207 hybrid_clock.cc:648] HybridClock initialized: now 1714723118594973 us; error 26 us; skew 500 ppm
I20240503 07:58:38.596632 3207 webserver.cc:460] Webserver started at http://127.0.99.133:34615/ using document root <none> and password file <none>
I20240503 07:58:38.597136 3207 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.597187 3207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.597337 3207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.598860 3207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b4316a9fd1e64780a693b466700774bd"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.599275 3207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b4316a9fd1e64780a693b466700774bd"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.602304 3207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:38.604281 3221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.605396 3207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:38.605499 3207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b4316a9fd1e64780a693b466700774bd"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.605583 3207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.616566 3207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.617149 3207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.617270 3207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.617859 3207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.618657 3207 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.618705 3207 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.618749 3207 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.618775 3207 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.626664 3207 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34145
I20240503 07:58:38.626755 3334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34145 every 8 connection(s)
I20240503 07:58:38.627519 3207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:38.635820 3335 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.636039 3335 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.636515 3335 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.637147 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3207
I20240503 07:58:38.637257 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:38.637511 2622 ts_manager.cc:194] Registered new tserver with Master: b4316a9fd1e64780a693b466700774bd (127.0.99.133:34145)
I20240503 07:58:38.638105 2622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:42323
I20240503 07:58:38.639338 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:38.743063 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.743263 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.743321 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.746368 3338 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.746433 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.746519 3338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:38.750025 3338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.750844 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.751945 3338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.760344 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.760810 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.761430 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.762166 3338 server_base.cc:1034] running on GCE node
I20240503 07:58:38.762519 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.763057 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.764227 3338 hybrid_clock.cc:648] HybridClock initialized: now 1714723118764207 us; error 37 us; skew 500 ppm
I20240503 07:58:38.766163 3338 webserver.cc:460] Webserver started at http://127.0.99.134:39829/ using document root <none> and password file <none>
I20240503 07:58:38.766687 3338 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.766744 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.766909 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.768483 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "377b7db17e524c0385a4cfbdd4803cf5"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.768978 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "377b7db17e524c0385a4cfbdd4803cf5"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.772197 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240503 07:58:38.774466 3352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.775424 3338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:38.775523 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "377b7db17e524c0385a4cfbdd4803cf5"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.775597 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.794271 3338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.794865 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.795007 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.795509 3338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.796334 3338 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.796388 3338 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.796437 3338 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.796461 3338 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.805023 3338 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33903
I20240503 07:58:38.805090 3465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33903 every 8 connection(s)
I20240503 07:58:38.805902 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:38.813671 3466 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.813861 3466 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.814313 3466 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.814723 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3338
I20240503 07:58:38.814831 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:58:38.815237 2623 ts_manager.cc:194] Registered new tserver with Master: 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134:33903)
I20240503 07:58:38.815779 2623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:42313
I20240503 07:58:38.816684 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:38.923943 3469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:38.924150 3469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:38.924193 3469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:38.927203 3469 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:38.927274 3469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:38.927348 3469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:38.930886 3469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:38.931725 3469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:38.933032 3469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:38.941671 3474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.942211 3475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:38.942891 3477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:38.943451 3469 server_base.cc:1034] running on GCE node
I20240503 07:58:38.943799 3469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:38.944337 3469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:38.945508 3469 hybrid_clock.cc:648] HybridClock initialized: now 1714723118945478 us; error 48 us; skew 500 ppm
I20240503 07:58:38.947367 3469 webserver.cc:460] Webserver started at http://127.0.99.135:38721/ using document root <none> and password file <none>
I20240503 07:58:38.947916 3469 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:38.947973 3469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:38.948145 3469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:38.949786 3469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "afda07f6bc6e420d81525f220d2441ce"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.950234 3469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "afda07f6bc6e420d81525f220d2441ce"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.953454 3469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:38.955514 3483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.956480 3469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:38.956574 3469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "afda07f6bc6e420d81525f220d2441ce"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:38.956653 3469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:38.961468 2811 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:38.971196 3469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:38.971815 3469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:38.972002 3469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:38.972493 3469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:38.973407 3469 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:38.973464 3469 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.973507 3469 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:38.973536 3469 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:38.982144 3469 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46165
I20240503 07:58:38.982170 3596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46165 every 8 connection(s)
I20240503 07:58:38.982976 3469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:38.991029 3597 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:38.991240 3597 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:38.991688 3597 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:38.992282 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3469
I20240503 07:58:38.992417 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:58:38.992622 2622 ts_manager.cc:194] Registered new tserver with Master: afda07f6bc6e420d81525f220d2441ce (127.0.99.135:46165)
I20240503 07:58:38.993167 2622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:57595
I20240503 07:58:38.994513 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:58:39.003409 2622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 07:58:39.019201 3269 tablet_service.cc:1467] Processing CreateTablet for tablet 85ac898d1f5344b8bb6dc00f9db27ae3 (DEFAULT_TABLE table=TestTable [id=e0eaaa1ebc314b83a300ca2d0ba1b24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:39.020077 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ac898d1f5344b8bb6dc00f9db27ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:39.024834 3611 tablet_bootstrap.cc:492] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Bootstrap starting.
I20240503 07:58:39.026508 3611 tablet_bootstrap.cc:654] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:39.027498 3611 log.cc:824] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:39.029028 3611 tablet_bootstrap.cc:492] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: No bootstrap required, opened a new log
I20240503 07:58:39.029197 3611 ts_tablet_manager.cc:1367] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:39.031903 3611 raft_consensus.cc:348] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:39.032059 3611 raft_consensus.cc:374] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:39.032104 3611 raft_consensus.cc:729] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Initialized, Role: FOLLOWER
I20240503 07:58:39.032455 3611 consensus_queue.cc:260] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:39.032563 3611 raft_consensus.cc:388] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:39.032631 3611 raft_consensus.cc:482] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:39.032727 3611 raft_consensus.cc:3037] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:39.101722 3611 raft_consensus.cc:504] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:39.102187 3611 leader_election.cc:302] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4316a9fd1e64780a693b466700774bd; no voters:
I20240503 07:58:39.102535 3611 leader_election.cc:288] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:39.102666 3613 raft_consensus.cc:2781] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:39.102916 3613 raft_consensus.cc:686] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 LEADER]: Becoming Leader. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Running, Role: LEADER
I20240503 07:58:39.103271 3335 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:39.103257 3613 consensus_queue.cc:237] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:39.103524 3611 ts_tablet_manager.cc:1398] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20240503 07:58:39.107324 2622 catalog_manager.cc:5484] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd reported cstate change: term changed from 0 to 1, leader changed from <none> to b4316a9fd1e64780a693b466700774bd (127.0.99.133). New cstate: current_term: 1 leader_uuid: "b4316a9fd1e64780a693b466700774bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:39.134204 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 07:58:39.135500 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9a340c22b2014883b6bf4e891be77b34 to finish bootstrapping
I20240503 07:58:39.141579 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c66f48bb18804160915381eac766f6fe to finish bootstrapping
I20240503 07:58:39.146375 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5294114bec9e421a866b4f999d3bb6bf to finish bootstrapping
I20240503 07:58:39.146721 2942 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:39.151279 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f9266aaeb8404667ae633cb22af9d372 to finish bootstrapping
I20240503 07:58:39.155977 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b4316a9fd1e64780a693b466700774bd to finish bootstrapping
I20240503 07:58:39.159855 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 377b7db17e524c0385a4cfbdd4803cf5 to finish bootstrapping
I20240503 07:58:39.164299 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver afda07f6bc6e420d81525f220d2441ce to finish bootstrapping
I20240503 07:58:39.168637 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2945
I20240503 07:58:39.173640 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3076
I20240503 07:58:39.178133 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3207
I20240503 07:58:39.182641 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3338
I20240503 07:58:39.186771 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3469
I20240503 07:58:42.942322 2622 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:42.954152 2745 tablet_service.cc:1467] Processing CreateTablet for tablet 5f714d0cb8cf4288ab1d64b851ea7e57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1889614fd8f04f61830e61faeac4c35f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:42.954156 2876 tablet_service.cc:1467] Processing CreateTablet for tablet ceab64ea8ffb4e33be8bc5c851b6119c (DEFAULT_TABLE table=rebalance_test_table_0 [id=1889614fd8f04f61830e61faeac4c35f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:42.954164 2744 tablet_service.cc:1467] Processing CreateTablet for tablet 0d27df72acf7499eb7b57aded052fc97 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1889614fd8f04f61830e61faeac4c35f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:42.955013 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d27df72acf7499eb7b57aded052fc97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:42.955013 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceab64ea8ffb4e33be8bc5c851b6119c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:42.955363 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f714d0cb8cf4288ab1d64b851ea7e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:42.959578 3633 tablet_bootstrap.cc:492] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:42.959579 3634 tablet_bootstrap.cc:492] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:42.961376 3634 tablet_bootstrap.cc:654] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:42.961514 3633 tablet_bootstrap.cc:654] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:42.962015 3634 log.cc:824] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:42.962071 3633 log.cc:824] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:42.963303 3633 tablet_bootstrap.cc:492] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:42.963304 3634 tablet_bootstrap.cc:492] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:42.963443 3634 ts_tablet_manager.cc:1367] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:58:42.963440 3633 ts_tablet_manager.cc:1367] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:58:42.966125 3633 raft_consensus.cc:348] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.966130 3634 raft_consensus.cc:348] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:42.966269 3633 raft_consensus.cc:374] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:42.966270 3634 raft_consensus.cc:374] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:42.966298 3633 raft_consensus.cc:729] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:42.966310 3634 raft_consensus.cc:729] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:42.966657 3633 consensus_queue.cc:260] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.966657 3634 consensus_queue.cc:260] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:42.966761 3633 raft_consensus.cc:388] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:42.966765 3634 raft_consensus.cc:388] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:42.966812 3633 raft_consensus.cc:482] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:42.966815 3634 raft_consensus.cc:482] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:42.966903 3634 raft_consensus.cc:3037] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:42.966905 3633 raft_consensus.cc:3037] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:42.967530 3633 raft_consensus.cc:504] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.967733 3634 raft_consensus.cc:504] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:42.967820 3633 leader_election.cc:302] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:42.968019 3634 leader_election.cc:302] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:42.968102 3633 leader_election.cc:288] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:42.968183 3637 raft_consensus.cc:2781] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:42.968266 3634 leader_election.cc:288] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:42.968340 3637 raft_consensus.cc:686] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:42.968585 3637 consensus_queue.cc:237] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.968906 3633 ts_tablet_manager.cc:1398] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:58:42.968962 3634 ts_tablet_manager.cc:1398] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:58:42.969130 3641 raft_consensus.cc:2781] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:42.969326 3641 raft_consensus.cc:686] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:42.969599 3633 tablet_bootstrap.cc:492] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:42.970386 3633 tablet_bootstrap.cc:654] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:42.970647 2622 catalog_manager.cc:5484] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:42.971359 3633 tablet_bootstrap.cc:492] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:42.971446 3633 ts_tablet_manager.cc:1367] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:42.970284 3641 consensus_queue.cc:237] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:42.971909 3633 raft_consensus.cc:348] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.971983 3633 raft_consensus.cc:374] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:42.972005 3633 raft_consensus.cc:729] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:42.972090 3633 consensus_queue.cc:260] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.972153 3633 raft_consensus.cc:388] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:42.972180 3633 raft_consensus.cc:482] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:42.972213 3633 raft_consensus.cc:3037] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:42.972970 3633 raft_consensus.cc:504] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.973088 3633 leader_election.cc:302] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:42.973171 3633 leader_election.cc:288] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:42.973378 3633 ts_tablet_manager.cc:1398] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:42.973904 3639 raft_consensus.cc:2781] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:42.973989 3639 raft_consensus.cc:686] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:42.974087 3639 consensus_queue.cc:237] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:42.974666 2622 catalog_manager.cc:5484] T ceab64ea8ffb4e33be8bc5c851b6119c P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:42.975219 2623 catalog_manager.cc:5484] T 0d27df72acf7499eb7b57aded052fc97 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.074604 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:43.074801 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:43.077697 3649 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 64.5094 ms
time per row: 0.0322547 ms
I20240503 07:58:43.179067 2621 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:43.183823 2874 tablet_service.cc:1467] Processing CreateTablet for tablet 18b0318923a944ae880627a8a0c7f7ab (DEFAULT_TABLE table=rebalance_test_table_1 [id=18194380838046e28eb77078fa1ca491]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.183930 2745 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=18194380838046e28eb77078fa1ca491]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.184149 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18b0318923a944ae880627a8a0c7f7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.184198 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.185757 3633 tablet_bootstrap.cc:492] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.183823 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 67633b2413ef4567bf371391a07772af (DEFAULT_TABLE table=rebalance_test_table_1 [id=18194380838046e28eb77078fa1ca491]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.185904 3634 tablet_bootstrap.cc:492] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.185993 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67633b2413ef4567bf371391a07772af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.186511 3633 tablet_bootstrap.cc:654] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.186801 3634 tablet_bootstrap.cc:654] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.187503 3633 tablet_bootstrap.cc:492] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.187580 3633 ts_tablet_manager.cc:1367] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.187841 3634 tablet_bootstrap.cc:492] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.187911 3634 ts_tablet_manager.cc:1367] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.187951 3633 raft_consensus.cc:348] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.188045 3633 raft_consensus.cc:374] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.188066 3633 raft_consensus.cc:729] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.188179 3633 consensus_queue.cc:260] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.188243 3633 raft_consensus.cc:388] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.188263 3633 raft_consensus.cc:482] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.188306 3633 raft_consensus.cc:3037] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.188401 3634 raft_consensus.cc:348] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.188477 3634 raft_consensus.cc:374] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.188504 3634 raft_consensus.cc:729] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.188611 3634 consensus_queue.cc:260] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.188681 3634 raft_consensus.cc:388] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.188719 3634 raft_consensus.cc:482] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.188761 3634 raft_consensus.cc:3037] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.189179 3633 raft_consensus.cc:504] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.189298 3633 leader_election.cc:302] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.189358 3634 raft_consensus.cc:504] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.189436 3633 leader_election.cc:288] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.189481 3639 raft_consensus.cc:2781] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.189563 3639 raft_consensus.cc:686] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.189679 3639 consensus_queue.cc:237] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.189750 3633 ts_tablet_manager.cc:1398] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.189810 3634 leader_election.cc:302] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.189942 3641 raft_consensus.cc:2781] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.190012 3641 raft_consensus.cc:686] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.190104 3641 consensus_queue.cc:237] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.190418 3634 leader_election.cc:288] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.190665 3634 ts_tablet_manager.cc:1398] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:58:43.190796 3634 tablet_bootstrap.cc:492] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.191484 2619 catalog_manager.cc:5484] T 18b0318923a944ae880627a8a0c7f7ab P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.191589 3634 tablet_bootstrap.cc:654] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.192400 2621 catalog_manager.cc:5484] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.193138 3634 tablet_bootstrap.cc:492] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.193214 3634 ts_tablet_manager.cc:1367] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:43.193606 3634 raft_consensus.cc:348] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.193689 3634 raft_consensus.cc:374] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.193719 3634 raft_consensus.cc:729] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.194084 3634 consensus_queue.cc:260] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.194170 3634 raft_consensus.cc:388] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.194202 3634 raft_consensus.cc:482] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.194236 3634 raft_consensus.cc:3037] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.194759 3634 raft_consensus.cc:504] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.194850 3634 leader_election.cc:302] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.194928 3634 leader_election.cc:288] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.194974 3641 raft_consensus.cc:2781] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.195083 3641 raft_consensus.cc:686] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.195160 3634 ts_tablet_manager.cc:1398] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:58:43.195184 3641 consensus_queue.cc:237] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.196870 2619 catalog_manager.cc:5484] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.296219 3691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:43.296407 3691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:43.299393 3691 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 72.4177 ms
time per row: 0.0362088 ms
I20240503 07:58:43.407276 2618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:43.412137 2745 tablet_service.cc:1467] Processing CreateTablet for tablet a146e960f86f4876881041ce1f33bafe (DEFAULT_TABLE table=rebalance_test_table_2 [id=f5f49e2e1a714bbeacb050f804d6e5aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.412235 2876 tablet_service.cc:1467] Processing CreateTablet for tablet b705940a75a541d9816297f580ed5b57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f5f49e2e1a714bbeacb050f804d6e5aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.412145 2744 tablet_service.cc:1467] Processing CreateTablet for tablet 79bc252e1a9c40778cb8c72ff8b8e662 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f5f49e2e1a714bbeacb050f804d6e5aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.412431 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a146e960f86f4876881041ce1f33bafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.412498 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b705940a75a541d9816297f580ed5b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.412680 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bc252e1a9c40778cb8c72ff8b8e662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.414238 3634 tablet_bootstrap.cc:492] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.414420 3633 tablet_bootstrap.cc:492] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.415134 3634 tablet_bootstrap.cc:654] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.415266 3633 tablet_bootstrap.cc:654] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.416236 3634 tablet_bootstrap.cc:492] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.416246 3633 tablet_bootstrap.cc:492] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.416326 3633 ts_tablet_manager.cc:1367] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.416352 3634 ts_tablet_manager.cc:1367] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.416797 3634 raft_consensus.cc:348] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.416797 3633 raft_consensus.cc:348] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.416908 3633 raft_consensus.cc:374] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.416908 3634 raft_consensus.cc:374] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.416951 3633 raft_consensus.cc:729] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.416951 3634 raft_consensus.cc:729] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.417078 3634 consensus_queue.cc:260] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.417078 3633 consensus_queue.cc:260] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.417158 3633 raft_consensus.cc:388] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.417160 3634 raft_consensus.cc:388] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.417200 3633 raft_consensus.cc:482] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.417200 3634 raft_consensus.cc:482] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.417245 3633 raft_consensus.cc:3037] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.417245 3634 raft_consensus.cc:3037] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.418025 3633 raft_consensus.cc:504] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.418025 3634 raft_consensus.cc:504] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.418166 3633 leader_election.cc:302] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.418166 3634 leader_election.cc:302] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.418265 3633 leader_election.cc:288] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.418267 3634 leader_election.cc:288] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.418313 3639 raft_consensus.cc:2781] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.418406 3639 raft_consensus.cc:686] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.418510 3641 raft_consensus.cc:2781] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.418504 3633 ts_tablet_manager.cc:1398] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.418586 3641 raft_consensus.cc:686] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.418514 3639 consensus_queue.cc:237] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.418664 3633 tablet_bootstrap.cc:492] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.418694 3641 consensus_queue.cc:237] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.419219 3634 ts_tablet_manager.cc:1398] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:58:43.419556 3633 tablet_bootstrap.cc:654] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.419919 2618 catalog_manager.cc:5484] T 79bc252e1a9c40778cb8c72ff8b8e662 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.420509 3633 tablet_bootstrap.cc:492] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.420564 3633 ts_tablet_manager.cc:1367] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:43.420984 3633 raft_consensus.cc:348] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.421072 3633 raft_consensus.cc:374] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.421105 3633 raft_consensus.cc:729] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.421240 3633 consensus_queue.cc:260] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.421327 3633 raft_consensus.cc:388] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.421363 3633 raft_consensus.cc:482] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.421404 3633 raft_consensus.cc:3037] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.421381 2616 catalog_manager.cc:5484] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.422031 3633 raft_consensus.cc:504] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.422120 3633 leader_election.cc:302] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.422190 3633 leader_election.cc:288] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.422248 3639 raft_consensus.cc:2781] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.422317 3639 raft_consensus.cc:686] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.422459 3633 ts_tablet_manager.cc:1398] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.422430 3639 consensus_queue.cc:237] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.423597 2618 catalog_manager.cc:5484] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.532604 3725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:43.532827 3725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:43.535712 3725 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 59.1009 ms
time per row: 0.0295504 ms
I20240503 07:58:43.631328 2616 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:43.636549 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 9d66454fdac8462ab56d5b23d4b80780 (DEFAULT_TABLE table=rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.636649 2874 tablet_service.cc:1467] Processing CreateTablet for tablet d8b7708dcbaf44178da851db2e62d702 (DEFAULT_TABLE table=rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.636670 2745 tablet_service.cc:1467] Processing CreateTablet for tablet 1602129ae4d042048001a370ffeb4d92 (DEFAULT_TABLE table=rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.636875 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d66454fdac8462ab56d5b23d4b80780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.636919 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1602129ae4d042048001a370ffeb4d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.637037 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b7708dcbaf44178da851db2e62d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.638630 3633 tablet_bootstrap.cc:492] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.639067 3634 tablet_bootstrap.cc:492] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.639449 3633 tablet_bootstrap.cc:654] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.639912 3634 tablet_bootstrap.cc:654] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.640677 3633 tablet_bootstrap.cc:492] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.640764 3633 ts_tablet_manager.cc:1367] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.640842 3634 tablet_bootstrap.cc:492] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.640921 3634 ts_tablet_manager.cc:1367] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:43.641209 3633 raft_consensus.cc:348] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.641302 3633 raft_consensus.cc:374] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.641336 3633 raft_consensus.cc:729] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.641289 3634 raft_consensus.cc:348] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.641388 3634 raft_consensus.cc:374] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.641429 3634 raft_consensus.cc:729] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.641469 3633 consensus_queue.cc:260] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.641532 3634 consensus_queue.cc:260] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.641557 3633 raft_consensus.cc:388] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.641597 3634 raft_consensus.cc:388] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.641595 3633 raft_consensus.cc:482] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.641636 3634 raft_consensus.cc:482] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.641649 3633 raft_consensus.cc:3037] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.641672 3634 raft_consensus.cc:3037] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.642318 3634 raft_consensus.cc:504] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.642355 3633 raft_consensus.cc:504] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.642432 3634 leader_election.cc:302] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.642486 3633 leader_election.cc:302] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.642606 3633 leader_election.cc:288] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.642647 3637 raft_consensus.cc:2781] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.642719 3637 raft_consensus.cc:686] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.642827 3637 consensus_queue.cc:237] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.642859 3634 leader_election.cc:288] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.642915 3638 raft_consensus.cc:2781] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.642987 3638 raft_consensus.cc:686] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.643110 3638 consensus_queue.cc:237] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.643296 3634 ts_tablet_manager.cc:1398] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.643107 3633 ts_tablet_manager.cc:1398] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.643822 3634 tablet_bootstrap.cc:492] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.644307 2616 catalog_manager.cc:5484] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.644361 2617 catalog_manager.cc:5484] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.644600 3634 tablet_bootstrap.cc:654] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.645650 3634 tablet_bootstrap.cc:492] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.645733 3634 ts_tablet_manager.cc:1367] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:43.646179 3634 raft_consensus.cc:348] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.646273 3634 raft_consensus.cc:374] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.646308 3634 raft_consensus.cc:729] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.646431 3634 consensus_queue.cc:260] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.646512 3634 raft_consensus.cc:388] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.646548 3634 raft_consensus.cc:482] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.646582 3634 raft_consensus.cc:3037] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.647224 3634 raft_consensus.cc:504] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.647365 3634 leader_election.cc:302] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.647464 3634 leader_election.cc:288] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.647518 3642 raft_consensus.cc:2781] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.647608 3642 raft_consensus.cc:686] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.647686 3634 ts_tablet_manager.cc:1398] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:43.647717 3642 consensus_queue.cc:237] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.650746 2621 catalog_manager.cc:5484] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.753181 3757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:43.753386 3757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:43.756273 3757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 57.8747 ms
time per row: 0.0289374 ms
I20240503 07:58:43.851295 2618 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32994:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:43.855954 2745 tablet_service.cc:1467] Processing CreateTablet for tablet aaba11f99cc843d2b67cdc46fa666350 (DEFAULT_TABLE table=rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.856016 2876 tablet_service.cc:1467] Processing CreateTablet for tablet 974d562e7c66487a8f767b3482a2daa6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.856225 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaba11f99cc843d2b67cdc46fa666350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.856412 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974d562e7c66487a8f767b3482a2daa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.857921 3633 tablet_bootstrap.cc:492] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.858217 3634 tablet_bootstrap.cc:492] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Bootstrap starting.
I20240503 07:58:43.858747 2745 tablet_service.cc:1467] Processing CreateTablet for tablet bd25b211a6b54c1fa079b799a8e6af9d (DEFAULT_TABLE table=rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:43.858865 3633 tablet_bootstrap.cc:654] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.858958 2745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd25b211a6b54c1fa079b799a8e6af9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:43.858959 3634 tablet_bootstrap.cc:654] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.859879 3633 tablet_bootstrap.cc:492] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.859923 3634 tablet_bootstrap.cc:492] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: No bootstrap required, opened a new log
I20240503 07:58:43.859963 3633 ts_tablet_manager.cc:1367] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:43.859978 3634 ts_tablet_manager.cc:1367] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:58:43.860350 3634 raft_consensus.cc:348] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.860352 3633 raft_consensus.cc:348] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.860442 3634 raft_consensus.cc:374] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.860442 3633 raft_consensus.cc:374] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.860471 3633 raft_consensus.cc:729] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.860472 3634 raft_consensus.cc:729] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.860579 3633 consensus_queue.cc:260] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.860579 3634 consensus_queue.cc:260] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.860654 3634 raft_consensus.cc:388] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.860654 3633 raft_consensus.cc:388] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.860702 3633 raft_consensus.cc:482] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.860702 3634 raft_consensus.cc:482] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.860741 3634 raft_consensus.cc:3037] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.860741 3633 raft_consensus.cc:3037] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.861819 3633 raft_consensus.cc:504] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.861819 3634 raft_consensus.cc:504] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.861944 3634 leader_election.cc:302] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:43.861944 3633 leader_election.cc:302] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.862022 3633 leader_election.cc:288] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.862041 3634 leader_election.cc:288] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.862089 3641 raft_consensus.cc:2781] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.862175 3641 raft_consensus.cc:686] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Leader. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:43.862229 3634 ts_tablet_manager.cc:1398] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:43.862237 3633 ts_tablet_manager.cc:1398] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:43.862267 3641 consensus_queue.cc:237] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } }
I20240503 07:58:43.862370 3633 tablet_bootstrap.cc:492] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Bootstrap starting.
I20240503 07:58:43.863112 3633 tablet_bootstrap.cc:654] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:43.863425 2618 catalog_manager.cc:5484] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe reported cstate change: term changed from 0 to 1, leader changed from <none> to c66f48bb18804160915381eac766f6fe (127.0.99.130). New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.863636 3640 raft_consensus.cc:2781] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.863744 3640 raft_consensus.cc:686] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.863957 3640 consensus_queue.cc:237] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.864598 3633 tablet_bootstrap.cc:492] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: No bootstrap required, opened a new log
I20240503 07:58:43.864673 3633 ts_tablet_manager.cc:1367] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:43.865089 3633 raft_consensus.cc:348] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.865187 3633 raft_consensus.cc:374] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:43.865226 3633 raft_consensus.cc:729] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Initialized, Role: FOLLOWER
I20240503 07:58:43.865234 2621 catalog_manager.cc:5484] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
I20240503 07:58:43.865327 3633 consensus_queue.cc:260] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.865408 3633 raft_consensus.cc:388] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:43.865447 3633 raft_consensus.cc:482] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:43.865491 3633 raft_consensus.cc:3037] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:43.866181 3633 raft_consensus.cc:504] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.866286 3633 leader_election.cc:302] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:43.866405 3637 raft_consensus.cc:2781] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:43.866477 3637 raft_consensus.cc:686] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Leader. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:43.866573 3637 consensus_queue.cc:237] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } }
I20240503 07:58:43.866915 3633 leader_election.cc:288] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:43.867161 3633 ts_tablet_manager.cc:1398] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:43.869263 2621 catalog_manager.cc:5484] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:43.972486 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:43.972681 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:43.975596 3787 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 54.6794 ms
time per row: 0.0273397 ms
I20240503 07:58:44.066274 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:40707
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:34925
--webserver_port=35457
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:44.167018 3814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:44.167194 3814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:44.167237 3814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:44.170238 3814 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:44.170296 3814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:44.170385 3814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:44.173804 3814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:40707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=35457
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:44.174505 3814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:44.175467 3814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:44.183316 3820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.183881 3814 server_base.cc:1034] running on GCE node
W20240503 07:58:44.184564 3822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.185190 3819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.185503 3814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:44.185977 3814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:44.187137 3814 hybrid_clock.cc:648] HybridClock initialized: now 1714723124187128 us; error 25 us; skew 500 ppm
I20240503 07:58:44.188832 3814 webserver.cc:460] Webserver started at http://127.0.99.131:35457/ using document root <none> and password file <none>
I20240503 07:58:44.189313 3814 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:44.189401 3814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:44.192377 3814 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:44.194103 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.194983 3814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:44.195071 3814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5294114bec9e421a866b4f999d3bb6bf"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:44.195457 3814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:44.209115 3814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:44.209620 3814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:44.209750 3814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:44.210201 3814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:44.210948 3814 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:44.210990 3814 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.211025 3814 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:44.211053 3814 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.218748 3814 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:40707
I20240503 07:58:44.218801 3941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:40707 every 8 connection(s)
I20240503 07:58:44.219646 3814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:44.220898 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3814
I20240503 07:58:44.221264 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:43805
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:34925
--webserver_port=40035
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:44.228552 3942 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:44.228790 3942 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:44.229368 3942 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:44.230540 2618 ts_manager.cc:194] Re-registered known tserver with Master: 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131:40707)
I20240503 07:58:44.231318 2618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:38947
W20240503 07:58:44.325583 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:44.325755 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:44.325794 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:44.328684 3945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:44.328738 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:44.328831 3945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:44.332211 3945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:43805
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=40035
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:44.332918 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:44.333859 3945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:44.341439 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.342121 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.342701 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.343276 3945 server_base.cc:1034] running on GCE node
I20240503 07:58:44.343571 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:44.344026 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:44.345173 3945 hybrid_clock.cc:648] HybridClock initialized: now 1714723124345157 us; error 32 us; skew 500 ppm
I20240503 07:58:44.346794 3945 webserver.cc:460] Webserver started at http://127.0.99.132:40035/ using document root <none> and password file <none>
I20240503 07:58:44.347273 3945 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:44.347342 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:44.350296 3945 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:44.352051 3959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.353067 3945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:44.353161 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f9266aaeb8404667ae633cb22af9d372"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:44.353529 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:44.364708 3945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:44.365243 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:44.365371 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:44.365816 3945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:44.366578 3945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:44.366629 3945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.366675 3945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:44.366699 3945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.374308 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43805
I20240503 07:58:44.374337 4072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43805 every 8 connection(s)
I20240503 07:58:44.375167 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:44.376071 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 3945
I20240503 07:58:44.376470 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34145
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:34925
--webserver_port=34615
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:44.383908 4073 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:44.384105 4073 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:44.384553 4073 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:44.385541 2618 ts_manager.cc:194] Re-registered known tserver with Master: f9266aaeb8404667ae633cb22af9d372 (127.0.99.132:43805)
I20240503 07:58:44.386054 2618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:58787
W20240503 07:58:44.478857 4076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:44.479022 4076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:44.479066 4076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:44.481992 4076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:44.482045 4076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:44.482113 4076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:44.485531 4076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34145
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=34615
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.4076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:44.486261 4076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:44.487242 4076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:44.495002 4084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.495654 4081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.496327 4082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.496915 4076 server_base.cc:1034] running on GCE node
I20240503 07:58:44.497253 4076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:44.497767 4076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:44.498921 4076 hybrid_clock.cc:648] HybridClock initialized: now 1714723124498904 us; error 42 us; skew 500 ppm
I20240503 07:58:44.500675 4076 webserver.cc:460] Webserver started at http://127.0.99.133:34615/ using document root <none> and password file <none>
I20240503 07:58:44.501190 4076 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:44.501262 4076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:44.504390 4076 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:44.506325 4090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.507300 4076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:44.507408 4076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b4316a9fd1e64780a693b466700774bd"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:44.507776 4076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:44.520449 4076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:44.520975 4076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:44.521106 4076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:44.521549 4076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:44.522540 4097 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 07:58:44.525251 4076 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 07:58:44.525307 4076 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240503 07:58:44.525362 4076 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 07:58:44.526636 4076 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 07:58:44.526675 4076 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:44.526887 4097 tablet_bootstrap.cc:492] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Bootstrap starting.
I20240503 07:58:44.535401 4076 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34145
I20240503 07:58:44.535434 4204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34145 every 8 connection(s)
I20240503 07:58:44.536227 4076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:44.539881 4097 log.cc:824] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:44.541472 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 4076
I20240503 07:58:44.541821 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:33903
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:34925
--webserver_port=39829
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:44.542183 4097 tablet_bootstrap.cc:492] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:44.542501 4097 tablet_bootstrap.cc:492] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Bootstrap complete.
I20240503 07:58:44.543534 4097 ts_tablet_manager.cc:1367] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240503 07:58:44.544409 4205 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:44.544620 4205 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:44.545284 4205 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:44.546108 4097 raft_consensus.cc:348] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:44.546399 4097 raft_consensus.cc:729] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Initialized, Role: FOLLOWER
I20240503 07:58:44.546787 4097 consensus_queue.cc:260] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:44.546918 4097 raft_consensus.cc:388] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:44.546977 4097 raft_consensus.cc:482] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:44.547065 4097 raft_consensus.cc:3037] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:44.547140 2618 ts_manager.cc:194] Re-registered known tserver with Master: b4316a9fd1e64780a693b466700774bd (127.0.99.133:34145)
I20240503 07:58:44.548092 2618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:50125
I20240503 07:58:44.548792 4097 raft_consensus.cc:504] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:44.549086 4097 leader_election.cc:302] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4316a9fd1e64780a693b466700774bd; no voters:
I20240503 07:58:44.549304 4097 leader_election.cc:288] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 07:58:44.549890 4097 ts_tablet_manager.cc:1398] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20240503 07:58:44.550191 4210 raft_consensus.cc:2781] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:44.550379 4210 raft_consensus.cc:686] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [term 2 LEADER]: Becoming Leader. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Running, Role: LEADER
I20240503 07:58:44.550410 4205 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:44.550659 4210 consensus_queue.cc:237] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } }
I20240503 07:58:44.552739 2618 catalog_manager.cc:5484] T 85ac898d1f5344b8bb6dc00f9db27ae3 P b4316a9fd1e64780a693b466700774bd reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b4316a9fd1e64780a693b466700774bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:44.644109 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:44.644284 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:44.644354 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:44.647290 4209 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:44.647362 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:44.647431 4209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:44.650902 4209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:33903
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=39829
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:44.651628 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:44.652644 4209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:44.660250 4222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.660728 4225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.661159 4223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.662009 4209 server_base.cc:1034] running on GCE node
I20240503 07:58:44.662314 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:44.662801 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:44.663945 4209 hybrid_clock.cc:648] HybridClock initialized: now 1714723124663934 us; error 33 us; skew 500 ppm
I20240503 07:58:44.665747 4209 webserver.cc:460] Webserver started at http://127.0.99.134:39829/ using document root <none> and password file <none>
I20240503 07:58:44.666237 4209 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:44.666311 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:44.669340 4209 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:58:44.671073 4231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.671963 4209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:44.672062 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "377b7db17e524c0385a4cfbdd4803cf5"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:44.672413 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:44.686717 4209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:44.687229 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:44.687374 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:44.687820 4209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:44.688596 4209 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:44.688644 4209 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.688694 4209 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:44.688727 4209 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.697811 4209 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33903
I20240503 07:58:44.697847 4344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33903 every 8 connection(s)
I20240503 07:58:44.698660 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:44.706365 4345 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:44.706554 4345 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:44.706960 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 4209
I20240503 07:58:44.707037 4345 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:44.707300 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:46165
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:34925
--webserver_port=38721
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:44.707899 2618 ts_manager.cc:194] Re-registered known tserver with Master: 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134:33903)
I20240503 07:58:44.708505 2618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:53025
W20240503 07:58:44.812067 4348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:44.812258 4348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:44.812320 4348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:44.815222 4348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:44.815277 4348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:44.815346 4348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:44.818907 4348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:46165
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=38721
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.4348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:44.819629 4348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:44.820613 4348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:44.828015 4353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.828478 4356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:44.829202 4354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:44.829918 4348 server_base.cc:1034] running on GCE node
I20240503 07:58:44.830225 4348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:44.830677 4348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:44.831820 4348 hybrid_clock.cc:648] HybridClock initialized: now 1714723124831805 us; error 28 us; skew 500 ppm
I20240503 07:58:44.833446 4348 webserver.cc:460] Webserver started at http://127.0.99.135:38721/ using document root <none> and password file <none>
I20240503 07:58:44.833900 4348 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:44.833962 4348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:44.836762 4348 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:44.838421 4362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.839262 4348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:44.839344 4348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "afda07f6bc6e420d81525f220d2441ce"
format_stamp: "Formatted at 2024-05-03 07:58:38 on dist-test-slave-7tt0"
I20240503 07:58:44.839699 4348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:44.860956 4348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:44.861455 4348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:44.861577 4348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:44.862002 4348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:44.862751 4348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:44.862793 4348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.862828 4348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:44.862855 4348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:44.871280 4348 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46165
I20240503 07:58:44.871308 4475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46165 every 8 connection(s)
I20240503 07:58:44.872123 4348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:44.872843 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 4348
I20240503 07:58:44.881620 4476 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:44.881799 4476 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:44.882225 4476 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:44.883039 2621 ts_manager.cc:194] Re-registered known tserver with Master: afda07f6bc6e420d81525f220d2441ce (127.0.99.135:46165)
I20240503 07:58:44.883563 2621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:42783
I20240503 07:58:45.058607 4410 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.060077 4279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.062546 3876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.063026 4007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.067926 4139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:45.069119 2745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:45.070097 2876 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:45.179814 2745 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:58:45.180166 4279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.184717 2876 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:58:45.190527 4410 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.190874 3876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.192328 4007 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:45.196456 4139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:58:45.233317 2896 consensus_queue.cc:237] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.233417 3942 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:45.234704 4567 raft_consensus.cc:2932] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } } }
I20240503 07:58:45.237195 2621 catalog_manager.cc:5484] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index -1 to 4, NON_VOTER f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.239147 2829 consensus_peers.cc:485] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe -> Peer f9266aaeb8404667ae633cb22af9d372 (127.0.99.132:43805): Couldn't send request to peer f9266aaeb8404667ae633cb22af9d372. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8b7708dcbaf44178da851db2e62d702. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.244040 2765 consensus_queue.cc:237] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.245215 4580 raft_consensus.cc:2932] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER afda07f6bc6e420d81525f220d2441ce (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } } }
I20240503 07:58:45.247604 2621 catalog_manager.cc:5484] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER afda07f6bc6e420d81525f220d2441ce (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.249540 2700 consensus_peers.cc:485] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 -> Peer afda07f6bc6e420d81525f220d2441ce (127.0.99.135:46165): Couldn't send request to peer afda07f6bc6e420d81525f220d2441ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd25b211a6b54c1fa079b799a8e6af9d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.253098 2765 consensus_queue.cc:237] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.253839 4579 raft_consensus.cc:2932] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } } }
I20240503 07:58:45.255811 2621 catalog_manager.cc:5484] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.257700 2699 consensus_peers.cc:485] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 -> Peer 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131:40707): Couldn't send request to peer 5294114bec9e421a866b4f999d3bb6bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a146e960f86f4876881041ce1f33bafe. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.262148 2896 consensus_queue.cc:237] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.262929 4567 raft_consensus.cc:2932] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } } }
I20240503 07:58:45.265362 2621 catalog_manager.cc:5484] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index -1 to 4, NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.266664 2829 consensus_peers.cc:485] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe -> Peer 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134:33903): Couldn't send request to peer 377b7db17e524c0385a4cfbdd4803cf5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67633b2413ef4567bf371391a07772af. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.271067 2896 consensus_queue.cc:237] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 07:58:45.271800 4568 raft_consensus.cc:2932] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4316a9fd1e64780a693b466700774bd (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } } }
I20240503 07:58:45.274215 2621 catalog_manager.cc:5484] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index -1 to 4, NON_VOTER b4316a9fd1e64780a693b466700774bd (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.275390 2829 consensus_peers.cc:485] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe -> Peer b4316a9fd1e64780a693b466700774bd (127.0.99.133:34145): Couldn't send request to peer b4316a9fd1e64780a693b466700774bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b705940a75a541d9816297f580ed5b57. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.280282 2765 consensus_queue.cc:237] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.281062 4580 raft_consensus.cc:2932] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } } }
I20240503 07:58:45.283046 2621 catalog_manager.cc:5484] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.284387 2698 consensus_peers.cc:485] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 -> Peer f9266aaeb8404667ae633cb22af9d372 (127.0.99.132:43805): Couldn't send request to peer f9266aaeb8404667ae633cb22af9d372. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f714d0cb8cf4288ab1d64b851ea7e57. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.288681 2896 consensus_queue.cc:237] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.289477 4568 raft_consensus.cc:2932] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER afda07f6bc6e420d81525f220d2441ce (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } } }
I20240503 07:58:45.291519 2621 catalog_manager.cc:5484] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index -1 to 4, NON_VOTER afda07f6bc6e420d81525f220d2441ce (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.293783 2831 consensus_peers.cc:485] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe -> Peer afda07f6bc6e420d81525f220d2441ce (127.0.99.135:46165): Couldn't send request to peer afda07f6bc6e420d81525f220d2441ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d66454fdac8462ab56d5b23d4b80780. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.297572 2765 consensus_queue.cc:237] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.298254 4579 raft_consensus.cc:2932] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } } }
W20240503 07:58:45.298734 2699 consensus_peers.cc:485] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 -> Peer 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131:40707): Couldn't send request to peer 5294114bec9e421a866b4f999d3bb6bf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ea6ab00e8f44f0888ecdb9a3084aac4. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.300125 2621 catalog_manager.cc:5484] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:45.305680 2765 consensus_queue.cc:237] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.306385 4579 raft_consensus.cc:2932] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } } }
I20240503 07:58:45.308459 2621 catalog_manager.cc:5484] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:45.310024 2698 consensus_peers.cc:485] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 -> Peer 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134:33903): Couldn't send request to peer 377b7db17e524c0385a4cfbdd4803cf5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaba11f99cc843d2b67cdc46fa666350. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:45.387777 4073 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:45.698057 4714 ts_tablet_manager.cc:902] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Initiating tablet copy from peer c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:45.698807 4714 tablet_copy_client.cc:308] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39115
I20240503 07:58:45.702893 2916 tablet_copy_service.cc:140] P c66f48bb18804160915381eac766f6fe: Received BeginTabletCopySession request for tablet d8b7708dcbaf44178da851db2e62d702 from peer f9266aaeb8404667ae633cb22af9d372 ({username='slave'} at 127.0.99.132:59739)
I20240503 07:58:45.703040 2916 tablet_copy_service.cc:161] P c66f48bb18804160915381eac766f6fe: Beginning new tablet copy session on tablet d8b7708dcbaf44178da851db2e62d702 from peer f9266aaeb8404667ae633cb22af9d372 at {username='slave'} at 127.0.99.132:59739: session id = f9266aaeb8404667ae633cb22af9d372-d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:45.704263 2916 tablet_copy_source_session.cc:205] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.706092 4720 ts_tablet_manager.cc:902] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Initiating tablet copy from peer c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:45.706815 4720 tablet_copy_client.cc:308] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39115
I20240503 07:58:45.707762 4714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8b7708dcbaf44178da851db2e62d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.710103 4345 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:45.710181 2916 tablet_copy_service.cc:140] P c66f48bb18804160915381eac766f6fe: Received BeginTabletCopySession request for tablet 9d66454fdac8462ab56d5b23d4b80780 from peer afda07f6bc6e420d81525f220d2441ce ({username='slave'} at 127.0.99.135:55289)
I20240503 07:58:45.710263 2916 tablet_copy_service.cc:161] P c66f48bb18804160915381eac766f6fe: Beginning new tablet copy session on tablet 9d66454fdac8462ab56d5b23d4b80780 from peer afda07f6bc6e420d81525f220d2441ce at {username='slave'} at 127.0.99.135:55289: session id = afda07f6bc6e420d81525f220d2441ce-9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:45.710884 2916 tablet_copy_source_session.cc:205] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.713779 4720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d66454fdac8462ab56d5b23d4b80780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.714217 4714 tablet_copy_client.cc:783] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.714406 4714 tablet_copy_client.cc:647] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.715950 4714 tablet_copy_client.cc:523] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.717264 4714 tablet_bootstrap.cc:492] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Bootstrap starting.
I20240503 07:58:45.718573 4720 tablet_copy_client.cc:783] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.718766 4720 tablet_copy_client.cc:647] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.720082 4720 tablet_copy_client.cc:523] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.721408 4720 tablet_bootstrap.cc:492] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Bootstrap starting.
I20240503 07:58:45.722275 4714 log.cc:824] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:45.725773 4720 log.cc:824] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:45.734903 4728 ts_tablet_manager.cc:902] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:45.735423 4728 tablet_copy_client.cc:308] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:45.739888 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet bd25b211a6b54c1fa079b799a8e6af9d from peer afda07f6bc6e420d81525f220d2441ce ({username='slave'} at 127.0.99.135:41523)
I20240503 07:58:45.740026 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet bd25b211a6b54c1fa079b799a8e6af9d from peer afda07f6bc6e420d81525f220d2441ce at {username='slave'} at 127.0.99.135:41523: session id = afda07f6bc6e420d81525f220d2441ce-bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:45.741238 2785 tablet_copy_source_session.cc:205] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.742240 4728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd25b211a6b54c1fa079b799a8e6af9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.744035 4728 tablet_copy_client.cc:783] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.744166 4728 tablet_copy_client.cc:647] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.745101 4714 tablet_bootstrap.cc:492] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.745405 4714 tablet_bootstrap.cc:492] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Bootstrap complete.
I20240503 07:58:45.745527 4728 tablet_copy_client.cc:523] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.745663 4714 ts_tablet_manager.cc:1367] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240503 07:58:45.746585 4728 tablet_bootstrap.cc:492] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Bootstrap starting.
I20240503 07:58:45.747390 4714 raft_consensus.cc:348] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.747567 4714 raft_consensus.cc:729] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9266aaeb8404667ae633cb22af9d372, State: Initialized, Role: LEARNER
I20240503 07:58:45.747871 4714 consensus_queue.cc:260] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.748023 4720 tablet_bootstrap.cc:492] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.748277 4720 tablet_bootstrap.cc:492] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Bootstrap complete.
I20240503 07:58:45.748513 4720 ts_tablet_manager.cc:1367] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.004s
I20240503 07:58:45.748596 4714 ts_tablet_manager.cc:1398] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:45.750075 2916 tablet_copy_service.cc:342] P c66f48bb18804160915381eac766f6fe: Request end of tablet copy session f9266aaeb8404667ae633cb22af9d372-d8b7708dcbaf44178da851db2e62d702 received from {username='slave'} at 127.0.99.132:59739
I20240503 07:58:45.750164 2916 tablet_copy_service.cc:433] P c66f48bb18804160915381eac766f6fe: ending tablet copy session f9266aaeb8404667ae633cb22af9d372-d8b7708dcbaf44178da851db2e62d702 on tablet d8b7708dcbaf44178da851db2e62d702 with peer f9266aaeb8404667ae633cb22af9d372
I20240503 07:58:45.751937 4720 raft_consensus.cc:348] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.752261 4720 raft_consensus.cc:729] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: afda07f6bc6e420d81525f220d2441ce, State: Initialized, Role: LEARNER
I20240503 07:58:45.752570 4720 consensus_queue.cc:260] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.753419 4720 ts_tablet_manager.cc:1398] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:58:45.753727 4476 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:45.754096 2916 tablet_copy_service.cc:342] P c66f48bb18804160915381eac766f6fe: Request end of tablet copy session afda07f6bc6e420d81525f220d2441ce-9d66454fdac8462ab56d5b23d4b80780 received from {username='slave'} at 127.0.99.135:55289
I20240503 07:58:45.754165 2916 tablet_copy_service.cc:433] P c66f48bb18804160915381eac766f6fe: ending tablet copy session afda07f6bc6e420d81525f220d2441ce-9d66454fdac8462ab56d5b23d4b80780 on tablet 9d66454fdac8462ab56d5b23d4b80780 with peer afda07f6bc6e420d81525f220d2441ce
I20240503 07:58:45.766692 4728 tablet_bootstrap.cc:492] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.766916 4728 tablet_bootstrap.cc:492] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Bootstrap complete.
I20240503 07:58:45.767048 4728 ts_tablet_manager.cc:1367] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240503 07:58:45.767421 4728 raft_consensus.cc:348] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.767501 4728 raft_consensus.cc:729] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: afda07f6bc6e420d81525f220d2441ce, State: Initialized, Role: LEARNER
I20240503 07:58:45.767571 4728 consensus_queue.cc:260] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: true } }
I20240503 07:58:45.767771 4728 ts_tablet_manager.cc:1398] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:45.768257 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session afda07f6bc6e420d81525f220d2441ce-bd25b211a6b54c1fa079b799a8e6af9d received from {username='slave'} at 127.0.99.135:41523
I20240503 07:58:45.768353 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session afda07f6bc6e420d81525f220d2441ce-bd25b211a6b54c1fa079b799a8e6af9d on tablet bd25b211a6b54c1fa079b799a8e6af9d with peer afda07f6bc6e420d81525f220d2441ce
I20240503 07:58:45.770272 4736 ts_tablet_manager.cc:902] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:45.770913 4736 tablet_copy_client.cc:308] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:45.773895 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet a146e960f86f4876881041ce1f33bafe from peer 5294114bec9e421a866b4f999d3bb6bf ({username='slave'} at 127.0.99.131:51471)
I20240503 07:58:45.773983 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet a146e960f86f4876881041ce1f33bafe from peer 5294114bec9e421a866b4f999d3bb6bf at {username='slave'} at 127.0.99.131:51471: session id = 5294114bec9e421a866b4f999d3bb6bf-a146e960f86f4876881041ce1f33bafe
I20240503 07:58:45.774569 2785 tablet_copy_source_session.cc:205] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.777251 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a146e960f86f4876881041ce1f33bafe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.781786 4736 tablet_copy_client.cc:783] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.781960 4736 tablet_copy_client.cc:647] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.783133 4736 tablet_copy_client.cc:523] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.783757 4742 ts_tablet_manager.cc:902] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Initiating tablet copy from peer c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:45.784574 4742 tablet_copy_client.cc:308] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39115
I20240503 07:58:45.784911 4736 tablet_bootstrap.cc:492] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap starting.
I20240503 07:58:45.787542 4714 ts_tablet_manager.cc:902] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:45.787935 4714 tablet_copy_client.cc:308] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:45.789803 2916 tablet_copy_service.cc:140] P c66f48bb18804160915381eac766f6fe: Received BeginTabletCopySession request for tablet 67633b2413ef4567bf371391a07772af from peer 377b7db17e524c0385a4cfbdd4803cf5 ({username='slave'} at 127.0.99.134:54059)
I20240503 07:58:45.789903 2916 tablet_copy_service.cc:161] P c66f48bb18804160915381eac766f6fe: Beginning new tablet copy session on tablet 67633b2413ef4567bf371391a07772af from peer 377b7db17e524c0385a4cfbdd4803cf5 at {username='slave'} at 127.0.99.134:54059: session id = 377b7db17e524c0385a4cfbdd4803cf5-67633b2413ef4567bf371391a07772af
I20240503 07:58:45.790712 2916 tablet_copy_source_session.cc:205] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.791114 4736 log.cc:824] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:45.791731 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet 5f714d0cb8cf4288ab1d64b851ea7e57 from peer f9266aaeb8404667ae633cb22af9d372 ({username='slave'} at 127.0.99.132:40167)
I20240503 07:58:45.791807 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet 5f714d0cb8cf4288ab1d64b851ea7e57 from peer f9266aaeb8404667ae633cb22af9d372 at {username='slave'} at 127.0.99.132:40167: session id = f9266aaeb8404667ae633cb22af9d372-5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:45.792380 2785 tablet_copy_source_session.cc:205] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.793506 4714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f714d0cb8cf4288ab1d64b851ea7e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.793617 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67633b2413ef4567bf371391a07772af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.795121 4714 tablet_copy_client.cc:783] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.795241 4714 tablet_copy_client.cc:647] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.796130 4714 tablet_copy_client.cc:523] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.797217 4714 tablet_bootstrap.cc:492] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Bootstrap starting.
I20240503 07:58:45.800364 4742 tablet_copy_client.cc:783] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.800554 4742 tablet_copy_client.cc:647] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.801885 4742 tablet_copy_client.cc:523] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.803409 4742 tablet_bootstrap.cc:492] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap starting.
I20240503 07:58:45.811136 4742 log.cc:824] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:45.819522 4714 tablet_bootstrap.cc:492] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.819752 4714 tablet_bootstrap.cc:492] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Bootstrap complete.
I20240503 07:58:45.819909 4714 ts_tablet_manager.cc:1367] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.003s
I20240503 07:58:45.820384 4714 raft_consensus.cc:348] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.820508 4714 raft_consensus.cc:729] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9266aaeb8404667ae633cb22af9d372, State: Initialized, Role: LEARNER
I20240503 07:58:45.820605 4714 consensus_queue.cc:260] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: true } }
I20240503 07:58:45.820875 4714 ts_tablet_manager.cc:1398] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:58:45.821293 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session f9266aaeb8404667ae633cb22af9d372-5f714d0cb8cf4288ab1d64b851ea7e57 received from {username='slave'} at 127.0.99.132:40167
I20240503 07:58:45.821357 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session f9266aaeb8404667ae633cb22af9d372-5f714d0cb8cf4288ab1d64b851ea7e57 on tablet 5f714d0cb8cf4288ab1d64b851ea7e57 with peer f9266aaeb8404667ae633cb22af9d372
I20240503 07:58:45.824359 4775 ts_tablet_manager.cc:902] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:45.824899 4775 tablet_copy_client.cc:308] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:45.825273 4736 tablet_bootstrap.cc:492] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.825598 4736 tablet_bootstrap.cc:492] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap complete.
I20240503 07:58:45.825896 4736 ts_tablet_manager.cc:1367] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.004s
I20240503 07:58:45.826927 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 from peer 5294114bec9e421a866b4f999d3bb6bf ({username='slave'} at 127.0.99.131:51471)
I20240503 07:58:45.827013 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 from peer 5294114bec9e421a866b4f999d3bb6bf at {username='slave'} at 127.0.99.131:51471: session id = 5294114bec9e421a866b4f999d3bb6bf-2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:45.827771 2785 tablet_copy_source_session.cc:205] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.827838 4736 raft_consensus.cc:348] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.828028 4736 raft_consensus.cc:729] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5294114bec9e421a866b4f999d3bb6bf, State: Initialized, Role: LEARNER
I20240503 07:58:45.828375 4736 consensus_queue.cc:260] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.828586 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.829233 4736 ts_tablet_manager.cc:1398] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:45.831050 4775 tablet_copy_client.cc:783] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.831199 4775 tablet_copy_client.cc:647] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.831382 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session 5294114bec9e421a866b4f999d3bb6bf-a146e960f86f4876881041ce1f33bafe received from {username='slave'} at 127.0.99.131:51471
I20240503 07:58:45.831439 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session 5294114bec9e421a866b4f999d3bb6bf-a146e960f86f4876881041ce1f33bafe on tablet a146e960f86f4876881041ce1f33bafe with peer 5294114bec9e421a866b4f999d3bb6bf
I20240503 07:58:45.833225 4775 tablet_copy_client.cc:523] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.834281 4775 tablet_bootstrap.cc:492] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap starting.
I20240503 07:58:45.841112 4742 tablet_bootstrap.cc:492] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.841387 4742 tablet_bootstrap.cc:492] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap complete.
I20240503 07:58:45.841643 4742 ts_tablet_manager.cc:1367] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.004s
I20240503 07:58:45.843901 4742 raft_consensus.cc:348] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.844126 4742 raft_consensus.cc:729] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 377b7db17e524c0385a4cfbdd4803cf5, State: Initialized, Role: LEARNER
I20240503 07:58:45.844439 4742 consensus_queue.cc:260] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.845265 4742 ts_tablet_manager.cc:1398] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:45.845860 2916 tablet_copy_service.cc:342] P c66f48bb18804160915381eac766f6fe: Request end of tablet copy session 377b7db17e524c0385a4cfbdd4803cf5-67633b2413ef4567bf371391a07772af received from {username='slave'} at 127.0.99.134:54059
I20240503 07:58:45.845937 2916 tablet_copy_service.cc:433] P c66f48bb18804160915381eac766f6fe: ending tablet copy session 377b7db17e524c0385a4cfbdd4803cf5-67633b2413ef4567bf371391a07772af on tablet 67633b2413ef4567bf371391a07772af with peer 377b7db17e524c0385a4cfbdd4803cf5
I20240503 07:58:45.856740 4775 tablet_bootstrap.cc:492] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.857059 4775 tablet_bootstrap.cc:492] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Bootstrap complete.
I20240503 07:58:45.857192 4775 ts_tablet_manager.cc:1367] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.002s
I20240503 07:58:45.857652 4775 raft_consensus.cc:348] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.857745 4775 raft_consensus.cc:729] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5294114bec9e421a866b4f999d3bb6bf, State: Initialized, Role: LEARNER
I20240503 07:58:45.857810 4775 consensus_queue.cc:260] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: true } }
I20240503 07:58:45.858021 4775 ts_tablet_manager.cc:1398] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:45.858448 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session 5294114bec9e421a866b4f999d3bb6bf-2ea6ab00e8f44f0888ecdb9a3084aac4 received from {username='slave'} at 127.0.99.131:51471
I20240503 07:58:45.858525 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session 5294114bec9e421a866b4f999d3bb6bf-2ea6ab00e8f44f0888ecdb9a3084aac4 on tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 with peer 5294114bec9e421a866b4f999d3bb6bf
I20240503 07:58:45.881347 4808 ts_tablet_manager.cc:902] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Initiating tablet copy from peer c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:45.881989 4808 tablet_copy_client.cc:308] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39115
I20240503 07:58:45.885066 2916 tablet_copy_service.cc:140] P c66f48bb18804160915381eac766f6fe: Received BeginTabletCopySession request for tablet b705940a75a541d9816297f580ed5b57 from peer b4316a9fd1e64780a693b466700774bd ({username='slave'} at 127.0.99.133:39245)
I20240503 07:58:45.885569 2916 tablet_copy_service.cc:161] P c66f48bb18804160915381eac766f6fe: Beginning new tablet copy session on tablet b705940a75a541d9816297f580ed5b57 from peer b4316a9fd1e64780a693b466700774bd at {username='slave'} at 127.0.99.133:39245: session id = b4316a9fd1e64780a693b466700774bd-b705940a75a541d9816297f580ed5b57
I20240503 07:58:45.886287 2916 tablet_copy_source_session.cc:205] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.886626 4742 ts_tablet_manager.cc:902] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:45.887054 4742 tablet_copy_client.cc:308] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:45.887933 4808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b705940a75a541d9816297f580ed5b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.890494 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet aaba11f99cc843d2b67cdc46fa666350 from peer 377b7db17e524c0385a4cfbdd4803cf5 ({username='slave'} at 127.0.99.134:38655)
I20240503 07:58:45.890578 4808 tablet_copy_client.cc:783] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.890595 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet aaba11f99cc843d2b67cdc46fa666350 from peer 377b7db17e524c0385a4cfbdd4803cf5 at {username='slave'} at 127.0.99.134:38655: session id = 377b7db17e524c0385a4cfbdd4803cf5-aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:45.890723 4808 tablet_copy_client.cc:647] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.891299 2785 tablet_copy_source_session.cc:205] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:45.891983 4808 tablet_copy_client.cc:523] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.892051 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaba11f99cc843d2b67cdc46fa666350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:45.893056 4808 tablet_bootstrap.cc:492] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Bootstrap starting.
I20240503 07:58:45.893558 4742 tablet_copy_client.cc:783] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:45.893682 4742 tablet_copy_client.cc:647] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:45.894690 4742 tablet_copy_client.cc:523] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:45.895525 4742 tablet_bootstrap.cc:492] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap starting.
I20240503 07:58:45.911957 4808 tablet_bootstrap.cc:492] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.912160 4808 tablet_bootstrap.cc:492] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Bootstrap complete.
I20240503 07:58:45.912242 4742 tablet_bootstrap.cc:492] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:45.912295 4808 ts_tablet_manager.cc:1367] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20240503 07:58:45.912448 4742 tablet_bootstrap.cc:492] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Bootstrap complete.
I20240503 07:58:45.912568 4742 ts_tablet_manager.cc:1367] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.001s
I20240503 07:58:45.912732 4808 raft_consensus.cc:348] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 07:58:45.912858 4808 raft_consensus.cc:729] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Initialized, Role: LEARNER
I20240503 07:58:45.912961 4808 consensus_queue.cc:260] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: true } }
I20240503 07:58:45.913018 4742 raft_consensus.cc:348] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.913117 4742 raft_consensus.cc:729] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 377b7db17e524c0385a4cfbdd4803cf5, State: Initialized, Role: LEARNER
I20240503 07:58:45.913208 4742 consensus_queue.cc:260] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: true } }
I20240503 07:58:45.913260 4808 ts_tablet_manager.cc:1398] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:58:45.913523 4742 ts_tablet_manager.cc:1398] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 07:58:45.914031 2916 tablet_copy_service.cc:342] P c66f48bb18804160915381eac766f6fe: Request end of tablet copy session b4316a9fd1e64780a693b466700774bd-b705940a75a541d9816297f580ed5b57 received from {username='slave'} at 127.0.99.133:39245
I20240503 07:58:45.914099 2916 tablet_copy_service.cc:433] P c66f48bb18804160915381eac766f6fe: ending tablet copy session b4316a9fd1e64780a693b466700774bd-b705940a75a541d9816297f580ed5b57 on tablet b705940a75a541d9816297f580ed5b57 with peer b4316a9fd1e64780a693b466700774bd
I20240503 07:58:45.914680 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session 377b7db17e524c0385a4cfbdd4803cf5-aaba11f99cc843d2b67cdc46fa666350 received from {username='slave'} at 127.0.99.134:38655
I20240503 07:58:45.914757 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session 377b7db17e524c0385a4cfbdd4803cf5-aaba11f99cc843d2b67cdc46fa666350 on tablet aaba11f99cc843d2b67cdc46fa666350 with peer 377b7db17e524c0385a4cfbdd4803cf5
I20240503 07:58:46.176276 4027 raft_consensus.cc:1203] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.227174 4430 raft_consensus.cc:1203] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.248843 4430 raft_consensus.cc:1203] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.266875 4299 raft_consensus.cc:1203] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.289462 4027 raft_consensus.cc:1203] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.331219 4159 raft_consensus.cc:1203] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.359061 4299 raft_consensus.cc:1203] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.391551 3896 raft_consensus.cc:1203] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.442595 3896 raft_consensus.cc:1203] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:46.657548 4867 raft_consensus.cc:1050] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER afda07f6bc6e420d81525f220d2441ce to VOTER
I20240503 07:58:46.658113 4867 consensus_queue.cc:237] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:46.659312 4430 raft_consensus.cc:1261] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.659823 4866 consensus_queue.cc:1035] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.662264 4869 raft_consensus.cc:1050] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: attempting to promote NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 to VOTER
I20240503 07:58:46.662484 4867 raft_consensus.cc:2932] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:46.662755 4869 consensus_queue.cc:237] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:46.663019 4430 raft_consensus.cc:2932] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:46.664009 4299 raft_consensus.cc:1261] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Refusing update from remote peer c66f48bb18804160915381eac766f6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.664711 4868 consensus_queue.cc:1035] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.665776 2614 catalog_manager.cc:5484] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:46.667219 4869 raft_consensus.cc:2932] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:46.667667 4299 raft_consensus.cc:2932] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:46.669875 2617 catalog_manager.cc:5484] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:46.730378 4866 raft_consensus.cc:1050] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER f9266aaeb8404667ae633cb22af9d372 to VOTER
I20240503 07:58:46.730692 4866 consensus_queue.cc:237] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:46.731818 4027 raft_consensus.cc:1261] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.732192 4867 consensus_queue.cc:1035] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.734472 4866 raft_consensus.cc:2932] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:46.734938 4027 raft_consensus.cc:2932] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:46.736091 4568 raft_consensus.cc:1050] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: attempting to promote NON_VOTER afda07f6bc6e420d81525f220d2441ce to VOTER
I20240503 07:58:46.736366 4568 consensus_queue.cc:237] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:46.736626 2614 catalog_manager.cc:5484] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:46.737725 4430 raft_consensus.cc:1261] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 LEARNER]: Refusing update from remote peer c66f48bb18804160915381eac766f6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.738090 4568 consensus_queue.cc:1035] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.739830 4868 raft_consensus.cc:2932] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:46.740234 4430 raft_consensus.cc:2932] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:46.742022 4568 raft_consensus.cc:1050] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: attempting to promote NON_VOTER f9266aaeb8404667ae633cb22af9d372 to VOTER
I20240503 07:58:46.742208 2617 catalog_manager.cc:5484] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce reported cstate change: config changed from index 4 to 5, afda07f6bc6e420d81525f220d2441ce (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:46.742300 4568 consensus_queue.cc:237] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:46.743180 4027 raft_consensus.cc:1261] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 LEARNER]: Refusing update from remote peer c66f48bb18804160915381eac766f6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.743484 4568 consensus_queue.cc:1035] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.744967 4973 raft_consensus.cc:2932] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:46.745332 4027 raft_consensus.cc:2932] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:46.747028 2614 catalog_manager.cc:5484] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 reported cstate change: config changed from index 4 to 5, f9266aaeb8404667ae633cb22af9d372 (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:46.780570 4568 raft_consensus.cc:1050] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: attempting to promote NON_VOTER b4316a9fd1e64780a693b466700774bd to VOTER
I20240503 07:58:46.780853 4568 consensus_queue.cc:237] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 07:58:46.781781 4159 raft_consensus.cc:1261] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 LEARNER]: Refusing update from remote peer c66f48bb18804160915381eac766f6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.782146 4568 consensus_queue.cc:1035] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.783823 4868 raft_consensus.cc:2932] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4316a9fd1e64780a693b466700774bd (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 07:58:46.784247 4159 raft_consensus.cc:2932] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4316a9fd1e64780a693b466700774bd (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 07:58:46.785701 2617 catalog_manager.cc:5484] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index 4 to 5, b4316a9fd1e64780a693b466700774bd (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:46.803314 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd25b211a6b54c1fa079b799a8e6af9d"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52578
I20240503 07:58:46.803471 2765 raft_consensus.cc:595] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.816519 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8b7708dcbaf44178da851db2e62d702"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42452
I20240503 07:58:46.816655 2896 raft_consensus.cc:595] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.827253 4580 raft_consensus.cc:1050] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER 5294114bec9e421a866b4f999d3bb6bf to VOTER
I20240503 07:58:46.827555 4580 consensus_queue.cc:237] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:46.828646 3896 raft_consensus.cc:1261] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.829053 4866 consensus_queue.cc:1035] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.830269 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d66454fdac8462ab56d5b23d4b80780"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42478
I20240503 07:58:46.830380 2896 raft_consensus.cc:595] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.831748 4867 raft_consensus.cc:2932] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.832288 3896 raft_consensus.cc:2932] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.834425 2614 catalog_manager.cc:5484] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.840332 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67633b2413ef4567bf371391a07772af"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42484
I20240503 07:58:46.840425 2896 raft_consensus.cc:595] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.848964 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b705940a75a541d9816297f580ed5b57"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42498
I20240503 07:58:46.849058 2896 raft_consensus.cc:595] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.857970 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f714d0cb8cf4288ab1d64b851ea7e57"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52604
I20240503 07:58:46.858079 2765 raft_consensus.cc:595] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:46.902549 4580 raft_consensus.cc:1050] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER 5294114bec9e421a866b4f999d3bb6bf to VOTER
I20240503 07:58:46.902918 4580 consensus_queue.cc:237] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:46.904070 3896 raft_consensus.cc:1261] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.904536 4866 consensus_queue.cc:1035] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.906337 4580 raft_consensus.cc:2932] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.906824 3896 raft_consensus.cc:2932] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.908890 2617 catalog_manager.cc:5484] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: config changed from index 4 to 5, 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:46.911641 4866 raft_consensus.cc:1050] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER 377b7db17e524c0385a4cfbdd4803cf5 to VOTER
I20240503 07:58:46.911924 4866 consensus_queue.cc:237] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:46.912995 4299 raft_consensus.cc:1261] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:46.913322 4866 consensus_queue.cc:1035] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:46.914917 4580 raft_consensus.cc:2932] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:46.915316 4299 raft_consensus.cc:2932] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:46.917042 2614 catalog_manager.cc:5484] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index 4 to 5, 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.068949 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aaba11f99cc843d2b67cdc46fa666350"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52618
I20240503 07:58:47.069088 2765 raft_consensus.cc:595] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.078811 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd25b211a6b54c1fa079b799a8e6af9d"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52638
I20240503 07:58:47.078939 2765 raft_consensus.cc:595] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.088533 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a146e960f86f4876881041ce1f33bafe"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52664
I20240503 07:58:47.088677 2765 raft_consensus.cc:595] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.097975 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8b7708dcbaf44178da851db2e62d702"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42520
I20240503 07:58:47.098116 2896 raft_consensus.cc:595] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.103670 4868 raft_consensus.cc:979] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: : Instructing follower 377b7db17e524c0385a4cfbdd4803cf5 to start an election
I20240503 07:58:47.103816 4868 raft_consensus.cc:1067] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Signalling peer 377b7db17e524c0385a4cfbdd4803cf5 to start an election
I20240503 07:58:47.104401 4299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67633b2413ef4567bf371391a07772af"
dest_uuid: "377b7db17e524c0385a4cfbdd4803cf5"
from {username='slave'} at 127.0.99.130:58297
I20240503 07:58:47.104612 4299 raft_consensus.cc:482] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.104751 4299 raft_consensus.cc:3037] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.105618 4299 raft_consensus.cc:504] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.106388 4299 leader_election.cc:288] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [CANDIDATE]: Term 2 election: Requested vote from peers c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:47.107050 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67633b2413ef4567bf371391a07772af" candidate_uuid: "377b7db17e524c0385a4cfbdd4803cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c66f48bb18804160915381eac766f6fe"
I20240503 07:58:47.107204 2896 raft_consensus.cc:3032] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.107257 2896 raft_consensus.cc:729] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:47.107406 2896 consensus_queue.cc:260] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.107542 2896 raft_consensus.cc:3037] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.108296 2896 raft_consensus.cc:2454] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 377b7db17e524c0385a4cfbdd4803cf5 in term 2.
I20240503 07:58:47.108618 4235 leader_election.cc:302] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 377b7db17e524c0385a4cfbdd4803cf5, c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:47.108639 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ea6ab00e8f44f0888ecdb9a3084aac4"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52676
I20240503 07:58:47.108734 2765 raft_consensus.cc:595] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.108845 4797 raft_consensus.cc:2781] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.109011 4797 raft_consensus.cc:686] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 LEADER]: Becoming Leader. State: Replica: 377b7db17e524c0385a4cfbdd4803cf5, State: Running, Role: LEADER
I20240503 07:58:47.109251 4797 consensus_queue.cc:237] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.111047 2617 catalog_manager.cc:5484] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 reported cstate change: term changed from 1 to 2, leader changed from c66f48bb18804160915381eac766f6fe (127.0.99.130) to 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "377b7db17e524c0385a4cfbdd4803cf5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:58:47.117591 2607 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 67633b2413ef4567bf371391a07772af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 07:58:47.120471 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d66454fdac8462ab56d5b23d4b80780"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42542
I20240503 07:58:47.120568 2896 raft_consensus.cc:595] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.125515 4866 raft_consensus.cc:979] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower f9266aaeb8404667ae633cb22af9d372 to start an election
I20240503 07:58:47.125615 4866 raft_consensus.cc:1067] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer f9266aaeb8404667ae633cb22af9d372 to start an election
I20240503 07:58:47.126149 4027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f714d0cb8cf4288ab1d64b851ea7e57"
dest_uuid: "f9266aaeb8404667ae633cb22af9d372"
from {username='slave'} at 127.0.99.129:48825
I20240503 07:58:47.126319 4027 raft_consensus.cc:482] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.126416 4027 raft_consensus.cc:3037] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.127313 4027 raft_consensus.cc:504] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.128160 4027 leader_election.cc:288] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:47.128513 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f714d0cb8cf4288ab1d64b851ea7e57" candidate_uuid: "f9266aaeb8404667ae633cb22af9d372" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:47.128667 2765 raft_consensus.cc:3032] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.128710 2765 raft_consensus.cc:729] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:47.128949 2765 consensus_queue.cc:260] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.129127 2765 raft_consensus.cc:3037] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.129993 2765 raft_consensus.cc:2454] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9266aaeb8404667ae633cb22af9d372 in term 2.
I20240503 07:58:47.130349 3962 leader_election.cc:302] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34, f9266aaeb8404667ae633cb22af9d372; no voters:
I20240503 07:58:47.130559 4733 raft_consensus.cc:2781] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.130735 4733 raft_consensus.cc:686] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 2 LEADER]: Becoming Leader. State: Replica: f9266aaeb8404667ae633cb22af9d372, State: Running, Role: LEADER
I20240503 07:58:47.130970 4733 consensus_queue.cc:237] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.132704 2617 catalog_manager.cc:5484] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to f9266aaeb8404667ae633cb22af9d372 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f9266aaeb8404667ae633cb22af9d372" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.136983 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b705940a75a541d9816297f580ed5b57"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42554
I20240503 07:58:47.137082 2896 raft_consensus.cc:595] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.163710 4973 raft_consensus.cc:979] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: : Instructing follower f9266aaeb8404667ae633cb22af9d372 to start an election
I20240503 07:58:47.163825 4973 raft_consensus.cc:1067] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Signalling peer f9266aaeb8404667ae633cb22af9d372 to start an election
I20240503 07:58:47.164258 4027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8b7708dcbaf44178da851db2e62d702"
dest_uuid: "f9266aaeb8404667ae633cb22af9d372"
from {username='slave'} at 127.0.99.130:55129
I20240503 07:58:47.164382 4027 raft_consensus.cc:482] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.164439 4027 raft_consensus.cc:3037] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.165256 4027 raft_consensus.cc:504] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.165593 4027 leader_election.cc:288] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [CANDIDATE]: Term 2 election: Requested vote from peers c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:47.165966 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8b7708dcbaf44178da851db2e62d702" candidate_uuid: "f9266aaeb8404667ae633cb22af9d372" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c66f48bb18804160915381eac766f6fe"
I20240503 07:58:47.166074 2896 raft_consensus.cc:3032] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.166121 2896 raft_consensus.cc:729] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:47.166178 4868 raft_consensus.cc:979] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: : Instructing follower afda07f6bc6e420d81525f220d2441ce to start an election
I20240503 07:58:47.166271 4868 raft_consensus.cc:1067] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Signalling peer afda07f6bc6e420d81525f220d2441ce to start an election
I20240503 07:58:47.166261 2896 consensus_queue.cc:260] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.166415 2896 raft_consensus.cc:3037] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.166726 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d66454fdac8462ab56d5b23d4b80780"
dest_uuid: "afda07f6bc6e420d81525f220d2441ce"
from {username='slave'} at 127.0.99.130:46747
I20240503 07:58:47.166884 4430 raft_consensus.cc:482] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.166980 4430 raft_consensus.cc:3037] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.167220 2896 raft_consensus.cc:2454] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9266aaeb8404667ae633cb22af9d372 in term 2.
I20240503 07:58:47.167470 3963 leader_election.cc:302] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c66f48bb18804160915381eac766f6fe, f9266aaeb8404667ae633cb22af9d372; no voters:
I20240503 07:58:47.167678 4733 raft_consensus.cc:2781] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.167747 4733 raft_consensus.cc:686] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 2 LEADER]: Becoming Leader. State: Replica: f9266aaeb8404667ae633cb22af9d372, State: Running, Role: LEADER
I20240503 07:58:47.167793 4430 raft_consensus.cc:504] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.167884 4733 consensus_queue.cc:237] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.168728 4430 leader_election.cc:288] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [CANDIDATE]: Term 2 election: Requested vote from peers c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:47.169066 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d66454fdac8462ab56d5b23d4b80780" candidate_uuid: "afda07f6bc6e420d81525f220d2441ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c66f48bb18804160915381eac766f6fe"
I20240503 07:58:47.169167 2896 raft_consensus.cc:3032] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.169205 2896 raft_consensus.cc:729] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:47.169332 2896 consensus_queue.cc:260] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.169440 2617 catalog_manager.cc:5484] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 reported cstate change: term changed from 1 to 2, leader changed from c66f48bb18804160915381eac766f6fe (127.0.99.130) to f9266aaeb8404667ae633cb22af9d372 (127.0.99.132). New cstate: current_term: 2 leader_uuid: "f9266aaeb8404667ae633cb22af9d372" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.169569 2896 raft_consensus.cc:3037] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.170186 2896 raft_consensus.cc:2454] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afda07f6bc6e420d81525f220d2441ce in term 2.
I20240503 07:58:47.170533 4366 leader_election.cc:302] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: afda07f6bc6e420d81525f220d2441ce, c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:47.170814 4735 raft_consensus.cc:2781] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.170977 4735 raft_consensus.cc:686] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 2 LEADER]: Becoming Leader. State: Replica: afda07f6bc6e420d81525f220d2441ce, State: Running, Role: LEADER
I20240503 07:58:47.171239 4735 consensus_queue.cc:237] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.173403 2617 catalog_manager.cc:5484] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce reported cstate change: term changed from 1 to 2, leader changed from c66f48bb18804160915381eac766f6fe (127.0.99.130) to afda07f6bc6e420d81525f220d2441ce (127.0.99.135). New cstate: current_term: 2 leader_uuid: "afda07f6bc6e420d81525f220d2441ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.193766 4866 raft_consensus.cc:979] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower afda07f6bc6e420d81525f220d2441ce to start an election
I20240503 07:58:47.193874 4866 raft_consensus.cc:1067] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer afda07f6bc6e420d81525f220d2441ce to start an election
I20240503 07:58:47.194257 4430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd25b211a6b54c1fa079b799a8e6af9d"
dest_uuid: "afda07f6bc6e420d81525f220d2441ce"
from {username='slave'} at 127.0.99.129:43751
I20240503 07:58:47.194371 4430 raft_consensus.cc:482] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.194416 4430 raft_consensus.cc:3037] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.195111 4430 raft_consensus.cc:504] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.195428 4430 leader_election.cc:288] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:47.195791 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd25b211a6b54c1fa079b799a8e6af9d" candidate_uuid: "afda07f6bc6e420d81525f220d2441ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:47.195897 2765 raft_consensus.cc:3032] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.195930 2765 raft_consensus.cc:729] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:47.196053 2765 consensus_queue.cc:260] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.196190 2765 raft_consensus.cc:3037] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.196905 2765 raft_consensus.cc:2454] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afda07f6bc6e420d81525f220d2441ce in term 2.
I20240503 07:58:47.197154 4365 leader_election.cc:302] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9a340c22b2014883b6bf4e891be77b34, afda07f6bc6e420d81525f220d2441ce; no voters:
I20240503 07:58:47.197329 4735 raft_consensus.cc:2781] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.197415 4735 raft_consensus.cc:686] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 2 LEADER]: Becoming Leader. State: Replica: afda07f6bc6e420d81525f220d2441ce, State: Running, Role: LEADER
I20240503 07:58:47.197540 4735 consensus_queue.cc:237] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.198693 2617 catalog_manager.cc:5484] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to afda07f6bc6e420d81525f220d2441ce (127.0.99.135). New cstate: current_term: 2 leader_uuid: "afda07f6bc6e420d81525f220d2441ce" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.259124 4973 raft_consensus.cc:979] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: : Instructing follower b4316a9fd1e64780a693b466700774bd to start an election
I20240503 07:58:47.259264 4973 raft_consensus.cc:1067] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Signalling peer b4316a9fd1e64780a693b466700774bd to start an election
I20240503 07:58:47.259694 4159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b705940a75a541d9816297f580ed5b57"
dest_uuid: "b4316a9fd1e64780a693b466700774bd"
from {username='slave'} at 127.0.99.130:41335
I20240503 07:58:47.259853 4159 raft_consensus.cc:482] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.259928 4159 raft_consensus.cc:3037] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.260663 4159 raft_consensus.cc:504] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 07:58:47.261327 4159 leader_election.cc:288] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 2 election: Requested vote from peers c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:47.261662 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b705940a75a541d9816297f580ed5b57" candidate_uuid: "b4316a9fd1e64780a693b466700774bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c66f48bb18804160915381eac766f6fe"
I20240503 07:58:47.261759 2896 raft_consensus.cc:3032] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.261803 2896 raft_consensus.cc:729] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:47.261938 2896 consensus_queue.cc:260] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 07:58:47.262056 2896 raft_consensus.cc:3037] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.262784 2896 raft_consensus.cc:2454] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4316a9fd1e64780a693b466700774bd in term 2.
I20240503 07:58:47.263093 4094 leader_election.cc:302] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b4316a9fd1e64780a693b466700774bd, c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:47.263279 4820 raft_consensus.cc:2781] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.263444 4820 raft_consensus.cc:686] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 2 LEADER]: Becoming Leader. State: Replica: b4316a9fd1e64780a693b466700774bd, State: Running, Role: LEADER
I20240503 07:58:47.263661 4820 consensus_queue.cc:237] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 07:58:47.265249 2617 catalog_manager.cc:5484] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd reported cstate change: term changed from 1 to 2, leader changed from c66f48bb18804160915381eac766f6fe (127.0.99.130) to b4316a9fd1e64780a693b466700774bd (127.0.99.133). New cstate: current_term: 2 leader_uuid: "b4316a9fd1e64780a693b466700774bd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.309527 4580 raft_consensus.cc:979] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower 377b7db17e524c0385a4cfbdd4803cf5 to start an election
I20240503 07:58:47.309625 4580 raft_consensus.cc:1067] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer 377b7db17e524c0385a4cfbdd4803cf5 to start an election
I20240503 07:58:47.310019 4299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aaba11f99cc843d2b67cdc46fa666350"
dest_uuid: "377b7db17e524c0385a4cfbdd4803cf5"
from {username='slave'} at 127.0.99.129:56937
I20240503 07:58:47.310164 4299 raft_consensus.cc:482] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.310215 4299 raft_consensus.cc:3037] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.311036 4299 raft_consensus.cc:504] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.311405 4299 leader_election.cc:288] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:47.311807 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaba11f99cc843d2b67cdc46fa666350" candidate_uuid: "377b7db17e524c0385a4cfbdd4803cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:47.311933 2765 raft_consensus.cc:3032] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.311975 2765 raft_consensus.cc:729] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:47.312109 2765 consensus_queue.cc:260] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.312224 2765 raft_consensus.cc:3037] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.312968 2765 raft_consensus.cc:2454] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 377b7db17e524c0385a4cfbdd4803cf5 in term 2.
I20240503 07:58:47.313243 4234 leader_election.cc:302] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 377b7db17e524c0385a4cfbdd4803cf5, 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:47.313431 4797 raft_consensus.cc:2781] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.313546 4797 raft_consensus.cc:686] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 LEADER]: Becoming Leader. State: Replica: 377b7db17e524c0385a4cfbdd4803cf5, State: Running, Role: LEADER
I20240503 07:58:47.313664 4797 consensus_queue.cc:237] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.314985 2617 catalog_manager.cc:5484] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to 377b7db17e524c0385a4cfbdd4803cf5 (127.0.99.134). New cstate: current_term: 2 leader_uuid: "377b7db17e524c0385a4cfbdd4803cf5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.335095 4866 raft_consensus.cc:979] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower 5294114bec9e421a866b4f999d3bb6bf to start an election
I20240503 07:58:47.335203 4866 raft_consensus.cc:1067] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer 5294114bec9e421a866b4f999d3bb6bf to start an election
I20240503 07:58:47.335616 3896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a146e960f86f4876881041ce1f33bafe"
dest_uuid: "5294114bec9e421a866b4f999d3bb6bf"
from {username='slave'} at 127.0.99.129:53197
I20240503 07:58:47.335784 3896 raft_consensus.cc:482] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.335896 3896 raft_consensus.cc:3037] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.336618 3896 raft_consensus.cc:504] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.337471 3896 leader_election.cc:288] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:47.337821 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a146e960f86f4876881041ce1f33bafe" candidate_uuid: "5294114bec9e421a866b4f999d3bb6bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:47.337924 2765 raft_consensus.cc:3032] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.337967 2765 raft_consensus.cc:729] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:47.338096 2765 consensus_queue.cc:260] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.338210 2765 raft_consensus.cc:3037] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.338876 2765 raft_consensus.cc:2454] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5294114bec9e421a866b4f999d3bb6bf in term 2.
I20240503 07:58:47.339190 3831 leader_election.cc:302] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5294114bec9e421a866b4f999d3bb6bf, 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:47.339403 4915 raft_consensus.cc:2781] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.339565 4915 raft_consensus.cc:686] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 2 LEADER]: Becoming Leader. State: Replica: 5294114bec9e421a866b4f999d3bb6bf, State: Running, Role: LEADER
I20240503 07:58:47.339831 4915 consensus_queue.cc:237] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.341414 2617 catalog_manager.cc:5484] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131). New cstate: current_term: 2 leader_uuid: "5294114bec9e421a866b4f999d3bb6bf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.376739 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ea6ab00e8f44f0888ecdb9a3084aac4"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52688
I20240503 07:58:47.376893 2765 raft_consensus.cc:595] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:47.461091 4580 raft_consensus.cc:979] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower 5294114bec9e421a866b4f999d3bb6bf to start an election
I20240503 07:58:47.461194 4580 raft_consensus.cc:1067] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer 5294114bec9e421a866b4f999d3bb6bf to start an election
I20240503 07:58:47.461731 3896 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ea6ab00e8f44f0888ecdb9a3084aac4"
dest_uuid: "5294114bec9e421a866b4f999d3bb6bf"
from {username='slave'} at 127.0.99.129:53197
I20240503 07:58:47.461860 3896 raft_consensus.cc:482] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:47.461923 3896 raft_consensus.cc:3037] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.462742 3896 raft_consensus.cc:504] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.463022 3896 leader_election.cc:288] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:47.463356 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea6ab00e8f44f0888ecdb9a3084aac4" candidate_uuid: "5294114bec9e421a866b4f999d3bb6bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:47.463477 2765 raft_consensus.cc:3032] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:47.463510 2765 raft_consensus.cc:729] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:47.463615 2765 consensus_queue.cc:260] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.463716 2765 raft_consensus.cc:3037] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:47.464344 2765 raft_consensus.cc:2454] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5294114bec9e421a866b4f999d3bb6bf in term 2.
I20240503 07:58:47.464627 3831 leader_election.cc:302] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5294114bec9e421a866b4f999d3bb6bf, 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:47.464881 4915 raft_consensus.cc:2781] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:47.464970 4915 raft_consensus.cc:686] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 2 LEADER]: Becoming Leader. State: Replica: 5294114bec9e421a866b4f999d3bb6bf, State: Running, Role: LEADER
I20240503 07:58:47.465113 4915 consensus_queue.cc:237] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.466542 2617 catalog_manager.cc:5484] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to 5294114bec9e421a866b4f999d3bb6bf (127.0.99.131). New cstate: current_term: 2 leader_uuid: "5294114bec9e421a866b4f999d3bb6bf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.555627 2896 raft_consensus.cc:1261] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Refusing update from remote peer 377b7db17e524c0385a4cfbdd4803cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.556175 4797 consensus_queue.cc:1035] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.602372 2896 raft_consensus.cc:1261] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Refusing update from remote peer afda07f6bc6e420d81525f220d2441ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.602852 4735 consensus_queue.cc:1035] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.605175 2765 raft_consensus.cc:1261] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer afda07f6bc6e420d81525f220d2441ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.605672 4735 consensus_queue.cc:1035] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.628468 2896 raft_consensus.cc:1261] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Refusing update from remote peer f9266aaeb8404667ae633cb22af9d372: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.628952 4733 consensus_queue.cc:1035] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.653898 2896 raft_consensus.cc:1261] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Refusing update from remote peer b4316a9fd1e64780a693b466700774bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.654286 4820 consensus_queue.cc:1035] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.711516 2765 raft_consensus.cc:1261] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer f9266aaeb8404667ae633cb22af9d372: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.711951 5132 consensus_queue.cc:1035] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.750494 4027 consensus_queue.cc:237] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.751066 4733 raft_consensus.cc:2932] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:47.752226 2607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5f714d0cb8cf4288ab1d64b851ea7e57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.753002 2614 catalog_manager.cc:5484] T 5f714d0cb8cf4288ab1d64b851ea7e57 P f9266aaeb8404667ae633cb22af9d372 reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "f9266aaeb8404667ae633cb22af9d372" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.754767 2765 raft_consensus.cc:1261] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer 5294114bec9e421a866b4f999d3bb6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.755268 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f714d0cb8cf4288ab1d64b851ea7e57 with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:47.755267 5148 consensus_queue.cc:1035] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.755697 5251 tablet_replica.cc:330] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:47.755991 5251 raft_consensus.cc:2229] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.756443 5251 raft_consensus.cc:2258] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.756484 4299 consensus_queue.cc:237] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.757052 4797 raft_consensus.cc:2932] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:47.758014 5251 ts_tablet_manager.cc:1875] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.758414 2607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67633b2413ef4567bf371391a07772af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.759373 2617 catalog_manager.cc:5484] T 67633b2413ef4567bf371391a07772af P 377b7db17e524c0385a4cfbdd4803cf5 reported cstate change: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "377b7db17e524c0385a4cfbdd4803cf5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.760493 5251 ts_tablet_manager.cc:1888] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.760592 5251 log.cc:1196] T 5f714d0cb8cf4288ab1d64b851ea7e57 P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:47.761188 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet 5f714d0cb8cf4288ab1d64b851ea7e57 (table rebalance_test_table_0 [id=1889614fd8f04f61830e61faeac4c35f]) successfully deleted
I20240503 07:58:47.761559 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 67633b2413ef4567bf371391a07772af with delete_type TABLET_DATA_TOMBSTONED (TS c66f48bb18804160915381eac766f6fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42086
I20240503 07:58:47.761965 5253 tablet_replica.cc:330] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: stopping tablet replica
I20240503 07:58:47.762207 5253 raft_consensus.cc:2229] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.762368 5253 raft_consensus.cc:2258] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.763250 5253 ts_tablet_manager.cc:1875] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.764976 5253 ts_tablet_manager.cc:1888] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.765053 5253 log.cc:1196] T 67633b2413ef4567bf371391a07772af P c66f48bb18804160915381eac766f6fe: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/67633b2413ef4567bf371391a07772af
I20240503 07:58:47.765645 2610 catalog_manager.cc:4830] TS c66f48bb18804160915381eac766f6fe (127.0.99.130:39115): tablet 67633b2413ef4567bf371391a07772af (table rebalance_test_table_1 [id=18194380838046e28eb77078fa1ca491]) successfully deleted
I20240503 07:58:47.785353 4430 consensus_queue.cc:237] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.785859 4735 raft_consensus.cc:2932] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:47.787017 2609 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d66454fdac8462ab56d5b23d4b80780 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.787711 2617 catalog_manager.cc:5484] T 9d66454fdac8462ab56d5b23d4b80780 P afda07f6bc6e420d81525f220d2441ce reported cstate change: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "afda07f6bc6e420d81525f220d2441ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.789527 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d66454fdac8462ab56d5b23d4b80780 with delete_type TABLET_DATA_TOMBSTONED (TS c66f48bb18804160915381eac766f6fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42086
I20240503 07:58:47.789695 5253 tablet_replica.cc:330] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: stopping tablet replica
I20240503 07:58:47.789784 5253 raft_consensus.cc:2229] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.789875 5253 raft_consensus.cc:2258] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.790345 5253 ts_tablet_manager.cc:1875] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.791677 5253 ts_tablet_manager.cc:1888] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.791734 5253 log.cc:1196] T 9d66454fdac8462ab56d5b23d4b80780 P c66f48bb18804160915381eac766f6fe: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:47.792083 2610 catalog_manager.cc:4830] TS c66f48bb18804160915381eac766f6fe (127.0.99.130:39115): tablet 9d66454fdac8462ab56d5b23d4b80780 (table rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]) successfully deleted
I20240503 07:58:47.805626 4027 consensus_queue.cc:237] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } }
I20240503 07:58:47.806138 5132 raft_consensus.cc:2932] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } } }
I20240503 07:58:47.807152 2607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d8b7708dcbaf44178da851db2e62d702 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.807822 2617 catalog_manager.cc:5484] T d8b7708dcbaf44178da851db2e62d702 P f9266aaeb8404667ae633cb22af9d372 reported cstate change: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "f9266aaeb8404667ae633cb22af9d372" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f9266aaeb8404667ae633cb22af9d372" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43805 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.809634 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet d8b7708dcbaf44178da851db2e62d702 with delete_type TABLET_DATA_TOMBSTONED (TS c66f48bb18804160915381eac766f6fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42086
I20240503 07:58:47.809800 5253 tablet_replica.cc:330] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: stopping tablet replica
I20240503 07:58:47.809886 5253 raft_consensus.cc:2229] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.809958 5253 raft_consensus.cc:2258] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.810400 5253 ts_tablet_manager.cc:1875] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.811666 5253 ts_tablet_manager.cc:1888] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.811731 5253 log.cc:1196] T d8b7708dcbaf44178da851db2e62d702 P c66f48bb18804160915381eac766f6fe: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:47.812091 2610 catalog_manager.cc:4830] TS c66f48bb18804160915381eac766f6fe (127.0.99.130:39115): tablet d8b7708dcbaf44178da851db2e62d702 (table rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]) successfully deleted
I20240503 07:58:47.884160 4430 consensus_queue.cc:237] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } }
I20240503 07:58:47.884902 4735 raft_consensus.cc:2932] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } } }
I20240503 07:58:47.887349 2609 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bd25b211a6b54c1fa079b799a8e6af9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.887840 2617 catalog_manager.cc:5484] T bd25b211a6b54c1fa079b799a8e6af9d P afda07f6bc6e420d81525f220d2441ce reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "afda07f6bc6e420d81525f220d2441ce" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "afda07f6bc6e420d81525f220d2441ce" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.890334 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet bd25b211a6b54c1fa079b799a8e6af9d with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:47.890583 5251 tablet_replica.cc:330] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:47.890712 5251 raft_consensus.cc:2229] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.890827 5251 raft_consensus.cc:2258] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.891547 2765 raft_consensus.cc:1261] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer 377b7db17e524c0385a4cfbdd4803cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.891680 5251 ts_tablet_manager.cc:1875] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.891983 5109 consensus_queue.cc:1035] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.893653 5251 ts_tablet_manager.cc:1888] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.893733 5251 log.cc:1196] T bd25b211a6b54c1fa079b799a8e6af9d P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:47.894178 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet bd25b211a6b54c1fa079b799a8e6af9d (table rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]) successfully deleted
I20240503 07:58:47.906388 4159 consensus_queue.cc:237] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } }
I20240503 07:58:47.906970 5146 raft_consensus.cc:2932] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } } }
I20240503 07:58:47.907788 2765 raft_consensus.cc:1261] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer 5294114bec9e421a866b4f999d3bb6bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:47.908227 4915 consensus_queue.cc:1035] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:47.908612 2607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b705940a75a541d9816297f580ed5b57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.909734 2614 catalog_manager.cc:5484] T b705940a75a541d9816297f580ed5b57 P b4316a9fd1e64780a693b466700774bd reported cstate change: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "b4316a9fd1e64780a693b466700774bd" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4316a9fd1e64780a693b466700774bd" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 34145 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.912802 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet b705940a75a541d9816297f580ed5b57 with delete_type TABLET_DATA_TOMBSTONED (TS c66f48bb18804160915381eac766f6fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42086
I20240503 07:58:47.913040 5253 tablet_replica.cc:330] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: stopping tablet replica
I20240503 07:58:47.913168 5253 raft_consensus.cc:2229] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.913288 5253 raft_consensus.cc:2258] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.914036 5253 ts_tablet_manager.cc:1875] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.916043 5253 ts_tablet_manager.cc:1888] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.916123 5253 log.cc:1196] T b705940a75a541d9816297f580ed5b57 P c66f48bb18804160915381eac766f6fe: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b705940a75a541d9816297f580ed5b57
I20240503 07:58:47.916628 2610 catalog_manager.cc:4830] TS c66f48bb18804160915381eac766f6fe (127.0.99.130:39115): tablet b705940a75a541d9816297f580ed5b57 (table rebalance_test_table_2 [id=f5f49e2e1a714bbeacb050f804d6e5aa]) successfully deleted
I20240503 07:58:47.922451 4299 consensus_queue.cc:237] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } }
I20240503 07:58:47.923020 4797 raft_consensus.cc:2932] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } } }
I20240503 07:58:47.924230 2607 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aaba11f99cc843d2b67cdc46fa666350 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.925053 2614 catalog_manager.cc:5484] T aaba11f99cc843d2b67cdc46fa666350 P 377b7db17e524c0385a4cfbdd4803cf5 reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "377b7db17e524c0385a4cfbdd4803cf5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "377b7db17e524c0385a4cfbdd4803cf5" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.927258 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet aaba11f99cc843d2b67cdc46fa666350 with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:47.927491 5251 tablet_replica.cc:330] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:47.927590 5251 raft_consensus.cc:2229] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.927680 5251 raft_consensus.cc:2258] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.928293 5251 ts_tablet_manager.cc:1875] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.930274 5251 ts_tablet_manager.cc:1888] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.930342 5251 log.cc:1196] T aaba11f99cc843d2b67cdc46fa666350 P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:47.930790 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet aaba11f99cc843d2b67cdc46fa666350 (table rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]) successfully deleted
I20240503 07:58:47.973618 3896 consensus_queue.cc:237] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:47.974129 5148 raft_consensus.cc:2932] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:47.975265 2608 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a146e960f86f4876881041ce1f33bafe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:47.976044 2614 catalog_manager.cc:5484] T a146e960f86f4876881041ce1f33bafe P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "5294114bec9e421a866b4f999d3bb6bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:47.978039 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet a146e960f86f4876881041ce1f33bafe with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:47.978243 5251 tablet_replica.cc:330] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:47.978355 5251 raft_consensus.cc:2229] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:47.978453 5251 raft_consensus.cc:2258] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:47.978978 5251 ts_tablet_manager.cc:1875] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:47.980437 5251 ts_tablet_manager.cc:1888] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:47.980496 5251 log.cc:1196] T a146e960f86f4876881041ce1f33bafe P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a146e960f86f4876881041ce1f33bafe
I20240503 07:58:47.980898 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet a146e960f86f4876881041ce1f33bafe (table rebalance_test_table_2 [id=f5f49e2e1a714bbeacb050f804d6e5aa]) successfully deleted
I20240503 07:58:48.090844 3896 consensus_queue.cc:237] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } }
I20240503 07:58:48.091463 5148 raft_consensus.cc:2932] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } } }
I20240503 07:58:48.092684 2608 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:48.093525 2614 catalog_manager.cc:5484] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 5294114bec9e421a866b4f999d3bb6bf reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "5294114bec9e421a866b4f999d3bb6bf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5294114bec9e421a866b4f999d3bb6bf" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 40707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:48.095650 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:48.095886 5251 tablet_replica.cc:330] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:48.096047 5251 raft_consensus.cc:2229] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:48.096153 5251 raft_consensus.cc:2258] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:48.096722 5251 ts_tablet_manager.cc:1875] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:48.098397 5251 ts_tablet_manager.cc:1888] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:48.098456 5251 log.cc:1196] T 2ea6ab00e8f44f0888ecdb9a3084aac4 P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:48.098850 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet 2ea6ab00e8f44f0888ecdb9a3084aac4 (table rebalance_test_table_1 [id=18194380838046e28eb77078fa1ca491]) successfully deleted
I20240503 07:58:48.206130 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.211887 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.212105 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.213191 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.214776 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.229189 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.239840 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.312755 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.314900 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.323967 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.324625 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.325150 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.326452 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.345714 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.541514 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.548332 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.548632 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.551613 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.554980 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.557211 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.568017 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.663820 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.664013 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.670773 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.674886 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.677122 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.686342 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.706552 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.791772 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.792918 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.794572 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.794797 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.797072 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:48.797226 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:48.804430 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
152cf5a0fb8c45389266683debe2a67e | 127.0.99.190:34925 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43981 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
377b7db17e524c0385a4cfbdd4803cf5 | 127.0.99.134:33903 | HEALTHY | <none> | 2 | 0
5294114bec9e421a866b4f999d3bb6bf | 127.0.99.131:40707 | HEALTHY | <none> | 2 | 0
9a340c22b2014883b6bf4e891be77b34 | 127.0.99.129:41775 | HEALTHY | <none> | 3 | 0
afda07f6bc6e420d81525f220d2441ce | 127.0.99.135:46165 | HEALTHY | <none> | 2 | 0
b4316a9fd1e64780a693b466700774bd | 127.0.99.133:34145 | HEALTHY | <none> | 2 | 0
c66f48bb18804160915381eac766f6fe | 127.0.99.130:39115 | HEALTHY | <none> | 3 | 0
f9266aaeb8404667ae633cb22af9d372 | 127.0.99.132:43805 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:41775
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:39115
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:40707
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43805
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:34145
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:33903
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:46165
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:41775
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:39115
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:40707
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43805
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:34145
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:33903
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:46165
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43981 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:58:48.840255 398 log_verifier.cc:126] Checking tablet 0d27df72acf7499eb7b57aded052fc97
I20240503 07:58:48.850421 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0d27df72acf7499eb7b57aded052fc97
I20240503 07:58:48.850487 398 log_verifier.cc:126] Checking tablet 1602129ae4d042048001a370ffeb4d92
I20240503 07:58:48.858784 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1602129ae4d042048001a370ffeb4d92
I20240503 07:58:48.858829 398 log_verifier.cc:126] Checking tablet 18b0318923a944ae880627a8a0c7f7ab
I20240503 07:58:48.866999 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 18b0318923a944ae880627a8a0c7f7ab
I20240503 07:58:48.867045 398 log_verifier.cc:126] Checking tablet 2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:48.875991 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:48.876046 398 log_verifier.cc:126] Checking tablet 5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:48.885665 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:48.885716 398 log_verifier.cc:126] Checking tablet 67633b2413ef4567bf371391a07772af
I20240503 07:58:48.894794 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67633b2413ef4567bf371391a07772af
I20240503 07:58:48.894845 398 log_verifier.cc:126] Checking tablet 79bc252e1a9c40778cb8c72ff8b8e662
I20240503 07:58:48.903535 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 79bc252e1a9c40778cb8c72ff8b8e662
I20240503 07:58:48.903587 398 log_verifier.cc:126] Checking tablet 85ac898d1f5344b8bb6dc00f9db27ae3
I20240503 07:58:48.911348 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 85ac898d1f5344b8bb6dc00f9db27ae3
I20240503 07:58:48.911398 398 log_verifier.cc:126] Checking tablet 974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:48.920253 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:48.920305 398 log_verifier.cc:126] Checking tablet 9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:48.929255 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:48.929316 398 log_verifier.cc:126] Checking tablet a146e960f86f4876881041ce1f33bafe
I20240503 07:58:48.938066 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a146e960f86f4876881041ce1f33bafe
I20240503 07:58:48.938122 398 log_verifier.cc:126] Checking tablet aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:48.948194 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:48.948243 398 log_verifier.cc:126] Checking tablet b705940a75a541d9816297f580ed5b57
I20240503 07:58:48.956900 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b705940a75a541d9816297f580ed5b57
I20240503 07:58:48.956948 398 log_verifier.cc:126] Checking tablet bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:48.965544 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:48.965597 398 log_verifier.cc:126] Checking tablet ceab64ea8ffb4e33be8bc5c851b6119c
I20240503 07:58:48.973939 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ceab64ea8ffb4e33be8bc5c851b6119c
I20240503 07:58:48.973989 398 log_verifier.cc:126] Checking tablet d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:48.982517 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:48.984252 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--builtin_ntp_servers=127.0.99.148:43981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:49.085183 5534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:49.085361 5534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:49.085404 5534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:49.088253 5534 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:49.088305 5534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:49.088366 5534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:49.091820 5534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:43981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34925
--never_fsync=true
--heap_profile_path=/tmp/kudu.5534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:49.092538 5534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:49.093588 5534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:49.101634 5542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:49.102105 5539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:49.102787 5540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:49.103477 5534 server_base.cc:1034] running on GCE node
I20240503 07:58:49.103823 5534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:49.104328 5534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:49.105476 5534 hybrid_clock.cc:648] HybridClock initialized: now 1714723129105468 us; error 28 us; skew 500 ppm
I20240503 07:58:49.107146 5534 webserver.cc:460] Webserver started at http://127.0.99.136:37361/ using document root <none> and password file <none>
I20240503 07:58:49.107641 5534 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:49.107695 5534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:49.107847 5534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:49.109335 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "794c7a3190fe4b0fafdde8f50af31b86"
format_stamp: "Formatted at 2024-05-03 07:58:49 on dist-test-slave-7tt0"
I20240503 07:58:49.109766 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "794c7a3190fe4b0fafdde8f50af31b86"
format_stamp: "Formatted at 2024-05-03 07:58:49 on dist-test-slave-7tt0"
I20240503 07:58:49.112919 5534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:49.114825 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:49.115746 5534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:49.115837 5534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "794c7a3190fe4b0fafdde8f50af31b86"
format_stamp: "Formatted at 2024-05-03 07:58:49 on dist-test-slave-7tt0"
I20240503 07:58:49.115912 5534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:49.129089 5534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:49.129581 5534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:49.129706 5534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:49.130132 5534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:49.130877 5534 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:49.130921 5534 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:49.130966 5534 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:49.130992 5534 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:49.141136 5534 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:43801
I20240503 07:58:49.142174 5534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:49.143509 5661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:43801 every 8 connection(s)
I20240503 07:58:49.149127 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 5534
I20240503 07:58:49.149235 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:58:49.150136 5662 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34925
I20240503 07:58:49.150322 5662 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:49.150763 5662 heartbeater.cc:511] Master 127.0.99.190:34925 requested a full tablet report, sending...
I20240503 07:58:49.151628 2621 ts_manager.cc:194] Registered new tserver with Master: 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136:43801)
I20240503 07:58:49.152249 2621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:41553
I20240503 07:58:49.324939 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.326727 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:49.328946 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.332195 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.335819 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:49.341326 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.349388 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.350051 5596 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:49.467491 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.467561 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.475661 5596 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:58:49.475847 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.477222 2745 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:49.479226 2876 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:58:49.481134 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.491552 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:49.525522 2896 consensus_queue.cc:237] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:49.526377 5755 raft_consensus.cc:2932] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } } }
I20240503 07:58:49.528884 2621 catalog_manager.cc:5484] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index -1 to 4, NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:49.530802 2830 consensus_peers.cc:485] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe -> Peer 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136:43801): Couldn't send request to peer 794c7a3190fe4b0fafdde8f50af31b86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 974d562e7c66487a8f767b3482a2daa6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:49.534569 2765 consensus_queue.cc:237] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:49.535669 5768 raft_consensus.cc:2932] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } } }
I20240503 07:58:49.537781 2621 catalog_manager.cc:5484] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index -1 to 4, NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:58:49.539680 2699 consensus_peers.cc:485] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 -> Peer 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136:43801): Couldn't send request to peer 794c7a3190fe4b0fafdde8f50af31b86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1602129ae4d042048001a370ffeb4d92. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:58:50.054728 5802 ts_tablet_manager.cc:902] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Initiating tablet copy from peer 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:50.055413 5802 tablet_copy_client.cc:308] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:41775
I20240503 07:58:50.059084 2785 tablet_copy_service.cc:140] P 9a340c22b2014883b6bf4e891be77b34: Received BeginTabletCopySession request for tablet 1602129ae4d042048001a370ffeb4d92 from peer 794c7a3190fe4b0fafdde8f50af31b86 ({username='slave'} at 127.0.99.136:45313)
I20240503 07:58:50.059192 2785 tablet_copy_service.cc:161] P 9a340c22b2014883b6bf4e891be77b34: Beginning new tablet copy session on tablet 1602129ae4d042048001a370ffeb4d92 from peer 794c7a3190fe4b0fafdde8f50af31b86 at {username='slave'} at 127.0.99.136:45313: session id = 794c7a3190fe4b0fafdde8f50af31b86-1602129ae4d042048001a370ffeb4d92
I20240503 07:58:50.059893 2785 tablet_copy_source_session.cc:205] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:50.062824 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1602129ae4d042048001a370ffeb4d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:50.067359 5802 tablet_copy_client.cc:783] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:50.067523 5802 tablet_copy_client.cc:647] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:50.068912 5802 tablet_copy_client.cc:523] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:50.070279 5802 tablet_bootstrap.cc:492] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap starting.
I20240503 07:58:50.074939 5802 log.cc:824] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:50.097312 5802 tablet_bootstrap.cc:492] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:50.097669 5802 tablet_bootstrap.cc:492] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap complete.
I20240503 07:58:50.097996 5802 ts_tablet_manager.cc:1367] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Time spent bootstrapping tablet: real 0.028s user 0.028s sys 0.000s
I20240503 07:58:50.100109 5802 raft_consensus.cc:348] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:50.100313 5802 raft_consensus.cc:729] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 794c7a3190fe4b0fafdde8f50af31b86, State: Initialized, Role: LEARNER
I20240503 07:58:50.100611 5802 consensus_queue.cc:260] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:50.101280 5662 heartbeater.cc:503] Master 127.0.99.190:34925 was elected leader, sending a full tablet report...
I20240503 07:58:50.101495 5802 ts_tablet_manager.cc:1398] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:50.102238 2785 tablet_copy_service.cc:342] P 9a340c22b2014883b6bf4e891be77b34: Request end of tablet copy session 794c7a3190fe4b0fafdde8f50af31b86-1602129ae4d042048001a370ffeb4d92 received from {username='slave'} at 127.0.99.136:45313
I20240503 07:58:50.102329 2785 tablet_copy_service.cc:433] P 9a340c22b2014883b6bf4e891be77b34: ending tablet copy session 794c7a3190fe4b0fafdde8f50af31b86-1602129ae4d042048001a370ffeb4d92 on tablet 1602129ae4d042048001a370ffeb4d92 with peer 794c7a3190fe4b0fafdde8f50af31b86
I20240503 07:58:50.150486 5802 ts_tablet_manager.cc:902] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Initiating tablet copy from peer c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:50.150971 5802 tablet_copy_client.cc:308] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:39115
I20240503 07:58:50.154270 2916 tablet_copy_service.cc:140] P c66f48bb18804160915381eac766f6fe: Received BeginTabletCopySession request for tablet 974d562e7c66487a8f767b3482a2daa6 from peer 794c7a3190fe4b0fafdde8f50af31b86 ({username='slave'} at 127.0.99.136:48995)
I20240503 07:58:50.154379 2916 tablet_copy_service.cc:161] P c66f48bb18804160915381eac766f6fe: Beginning new tablet copy session on tablet 974d562e7c66487a8f767b3482a2daa6 from peer 794c7a3190fe4b0fafdde8f50af31b86 at {username='slave'} at 127.0.99.136:48995: session id = 794c7a3190fe4b0fafdde8f50af31b86-974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:50.155076 2916 tablet_copy_source_session.cc:205] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:58:50.155825 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974d562e7c66487a8f767b3482a2daa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:50.157442 5802 tablet_copy_client.cc:783] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Starting download of 0 data blocks...
I20240503 07:58:50.157582 5802 tablet_copy_client.cc:647] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Starting download of 1 WAL segments...
I20240503 07:58:50.158551 5802 tablet_copy_client.cc:523] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:58:50.159407 5802 tablet_bootstrap.cc:492] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap starting.
I20240503 07:58:50.175423 5802 tablet_bootstrap.cc:492] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:58:50.175648 5802 tablet_bootstrap.cc:492] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Bootstrap complete.
I20240503 07:58:50.175906 5802 ts_tablet_manager.cc:1367] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.002s
I20240503 07:58:50.176448 5802 raft_consensus.cc:348] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:50.176544 5802 raft_consensus.cc:729] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 794c7a3190fe4b0fafdde8f50af31b86, State: Initialized, Role: LEARNER
I20240503 07:58:50.176633 5802 consensus_queue.cc:260] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: true } }
I20240503 07:58:50.176931 5802 ts_tablet_manager.cc:1398] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:50.177449 2916 tablet_copy_service.cc:342] P c66f48bb18804160915381eac766f6fe: Request end of tablet copy session 794c7a3190fe4b0fafdde8f50af31b86-974d562e7c66487a8f767b3482a2daa6 received from {username='slave'} at 127.0.99.136:48995
I20240503 07:58:50.177522 2916 tablet_copy_service.cc:433] P c66f48bb18804160915381eac766f6fe: ending tablet copy session 794c7a3190fe4b0fafdde8f50af31b86-974d562e7c66487a8f767b3482a2daa6 on tablet 974d562e7c66487a8f767b3482a2daa6 with peer 794c7a3190fe4b0fafdde8f50af31b86
I20240503 07:58:50.594250 5616 raft_consensus.cc:1203] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:50.611352 5616 raft_consensus.cc:1203] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:58:51.077368 5850 raft_consensus.cc:1050] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: attempting to promote NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 to VOTER
I20240503 07:58:51.077711 5850 consensus_queue.cc:237] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.078871 5616 raft_consensus.cc:1261] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Refusing update from remote peer c66f48bb18804160915381eac766f6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:51.079268 5850 consensus_queue.cc:1035] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:51.081775 5810 raft_consensus.cc:2932] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:51.082280 5616 raft_consensus.cc:2932] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:51.083964 2621 catalog_manager.cc:5484] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe reported cstate change: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c66f48bb18804160915381eac766f6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:51.096959 5877 raft_consensus.cc:1050] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: attempting to promote NON_VOTER 794c7a3190fe4b0fafdde8f50af31b86 to VOTER
I20240503 07:58:51.097244 5877 consensus_queue.cc:237] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.098157 5616 raft_consensus.cc:1261] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 LEARNER]: Refusing update from remote peer 9a340c22b2014883b6bf4e891be77b34: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:58:51.098539 5877 consensus_queue.cc:1035] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:58:51.100137 5878 raft_consensus.cc:2932] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:51.100600 5616 raft_consensus.cc:2932] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:51.102737 2623 catalog_manager.cc:5484] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 reported cstate change: config changed from index 4 to 5, 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a340c22b2014883b6bf4e891be77b34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:51.236280 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1602129ae4d042048001a370ffeb4d92"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46304
I20240503 07:58:51.236438 2765 raft_consensus.cc:595] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:51.245586 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974d562e7c66487a8f767b3482a2daa6"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39220
I20240503 07:58:51.245721 2896 raft_consensus.cc:595] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:51.456389 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1602129ae4d042048001a370ffeb4d92"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46336
I20240503 07:58:51.456543 2765 raft_consensus.cc:595] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:51.466394 2896 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974d562e7c66487a8f767b3482a2daa6"
dest_uuid: "c66f48bb18804160915381eac766f6fe"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39240
I20240503 07:58:51.466523 2896 raft_consensus.cc:595] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:51.600179 5922 raft_consensus.cc:979] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: : Instructing follower 794c7a3190fe4b0fafdde8f50af31b86 to start an election
I20240503 07:58:51.600277 5922 raft_consensus.cc:1067] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Signalling peer 794c7a3190fe4b0fafdde8f50af31b86 to start an election
I20240503 07:58:51.600733 5616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "974d562e7c66487a8f767b3482a2daa6"
dest_uuid: "794c7a3190fe4b0fafdde8f50af31b86"
from {username='slave'} at 127.0.99.130:36101
I20240503 07:58:51.600936 5616 raft_consensus.cc:482] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:51.601040 5616 raft_consensus.cc:3037] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:51.601837 5616 raft_consensus.cc:504] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.602700 5616 leader_election.cc:288] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [CANDIDATE]: Term 2 election: Requested vote from peers c66f48bb18804160915381eac766f6fe (127.0.99.130:39115)
I20240503 07:58:51.603044 2896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974d562e7c66487a8f767b3482a2daa6" candidate_uuid: "794c7a3190fe4b0fafdde8f50af31b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c66f48bb18804160915381eac766f6fe"
I20240503 07:58:51.603165 2896 raft_consensus.cc:3032] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:51.603209 2896 raft_consensus.cc:729] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c66f48bb18804160915381eac766f6fe, State: Running, Role: LEADER
I20240503 07:58:51.603334 2896 consensus_queue.cc:260] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.603464 2896 raft_consensus.cc:3037] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:51.604199 2896 raft_consensus.cc:2454] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c7a3190fe4b0fafdde8f50af31b86 in term 2.
I20240503 07:58:51.604538 5552 leader_election.cc:302] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 794c7a3190fe4b0fafdde8f50af31b86, c66f48bb18804160915381eac766f6fe; no voters:
I20240503 07:58:51.604844 5924 raft_consensus.cc:2781] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:51.604998 5924 raft_consensus.cc:686] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 LEADER]: Becoming Leader. State: Replica: 794c7a3190fe4b0fafdde8f50af31b86, State: Running, Role: LEADER
I20240503 07:58:51.605288 5924 consensus_queue.cc:237] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.606986 2621 catalog_manager.cc:5484] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 reported cstate change: term changed from 1 to 2, leader changed from c66f48bb18804160915381eac766f6fe (127.0.99.130) to 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "794c7a3190fe4b0fafdde8f50af31b86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:58:51.612669 2608 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 974d562e7c66487a8f767b3482a2daa6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136:43801) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 07:58:51.676155 2765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1602129ae4d042048001a370ffeb4d92"
dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46354
I20240503 07:58:51.676281 2765 raft_consensus.cc:595] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:58:51.722503 5946 raft_consensus.cc:979] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: : Instructing follower 794c7a3190fe4b0fafdde8f50af31b86 to start an election
I20240503 07:58:51.722592 5946 raft_consensus.cc:1067] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Signalling peer 794c7a3190fe4b0fafdde8f50af31b86 to start an election
I20240503 07:58:51.723045 5616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1602129ae4d042048001a370ffeb4d92"
dest_uuid: "794c7a3190fe4b0fafdde8f50af31b86"
from {username='slave'} at 127.0.99.129:37429
I20240503 07:58:51.723165 5616 raft_consensus.cc:482] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:58:51.723232 5616 raft_consensus.cc:3037] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:51.724206 5616 raft_consensus.cc:504] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.724581 5616 leader_election.cc:288] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [CANDIDATE]: Term 2 election: Requested vote from peers 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775)
I20240503 07:58:51.724931 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1602129ae4d042048001a370ffeb4d92" candidate_uuid: "794c7a3190fe4b0fafdde8f50af31b86" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a340c22b2014883b6bf4e891be77b34"
I20240503 07:58:51.725054 2765 raft_consensus.cc:3032] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:58:51.725119 2765 raft_consensus.cc:729] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a340c22b2014883b6bf4e891be77b34, State: Running, Role: LEADER
I20240503 07:58:51.725270 2765 consensus_queue.cc:260] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.725421 2765 raft_consensus.cc:3037] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:58:51.726316 2765 raft_consensus.cc:2454] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c7a3190fe4b0fafdde8f50af31b86 in term 2.
I20240503 07:58:51.726620 5551 leader_election.cc:302] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 794c7a3190fe4b0fafdde8f50af31b86, 9a340c22b2014883b6bf4e891be77b34; no voters:
I20240503 07:58:51.726768 5924 raft_consensus.cc:2781] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:58:51.726845 5924 raft_consensus.cc:686] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 LEADER]: Becoming Leader. State: Replica: 794c7a3190fe4b0fafdde8f50af31b86, State: Running, Role: LEADER
I20240503 07:58:51.726972 5924 consensus_queue.cc:237] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:51.728389 2621 catalog_manager.cc:5484] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 reported cstate change: term changed from 1 to 2, leader changed from 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) to 794c7a3190fe4b0fafdde8f50af31b86 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "794c7a3190fe4b0fafdde8f50af31b86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:52.069640 2896 raft_consensus.cc:1261] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Refusing update from remote peer 794c7a3190fe4b0fafdde8f50af31b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:52.070117 5924 consensus_queue.cc:1035] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c66f48bb18804160915381eac766f6fe" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:52.255615 5616 consensus_queue.cc:237] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:52.256379 5925 raft_consensus.cc:2932] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:52.259109 2608 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 974d562e7c66487a8f767b3482a2daa6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 07:58:52.259581 2623 catalog_manager.cc:5484] T 974d562e7c66487a8f767b3482a2daa6 P 794c7a3190fe4b0fafdde8f50af31b86 reported cstate change: config changed from index 5 to 7, VOTER c66f48bb18804160915381eac766f6fe (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "794c7a3190fe4b0fafdde8f50af31b86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:52.262319 2876 tablet_service.cc:1514] Processing DeleteTablet for tablet 974d562e7c66487a8f767b3482a2daa6 with delete_type TABLET_DATA_TOMBSTONED (TS c66f48bb18804160915381eac766f6fe not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42086
I20240503 07:58:52.262652 5973 tablet_replica.cc:330] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: stopping tablet replica
I20240503 07:58:52.262768 5973 raft_consensus.cc:2229] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:52.262871 5973 raft_consensus.cc:2258] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:52.263890 5973 ts_tablet_manager.cc:1875] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:52.265607 5973 ts_tablet_manager.cc:1888] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:52.265671 5973 log.cc:1196] T 974d562e7c66487a8f767b3482a2daa6 P c66f48bb18804160915381eac766f6fe: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:52.266139 2610 catalog_manager.cc:4830] TS c66f48bb18804160915381eac766f6fe (127.0.99.130:39115): tablet 974d562e7c66487a8f767b3482a2daa6 (table rebalance_test_table_4 [id=73e80f544ef047c99d0bb8db956872e9]) successfully deleted
I20240503 07:58:52.352200 2765 raft_consensus.cc:1261] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Refusing update from remote peer 794c7a3190fe4b0fafdde8f50af31b86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:58:52.352636 5924 consensus_queue.cc:1035] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a340c22b2014883b6bf4e891be77b34" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 41775 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:58:52.865466 5616 consensus_queue.cc:237] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } }
I20240503 07:58:52.866084 5925 raft_consensus.cc:2932] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } } }
I20240503 07:58:52.867496 2608 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1602129ae4d042048001a370ffeb4d92 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240503 07:58:52.868410 2621 catalog_manager.cc:5484] T 1602129ae4d042048001a370ffeb4d92 P 794c7a3190fe4b0fafdde8f50af31b86 reported cstate change: config changed from index 5 to 7, VOTER 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "794c7a3190fe4b0fafdde8f50af31b86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "794c7a3190fe4b0fafdde8f50af31b86" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 43801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:58:52.870797 2745 tablet_service.cc:1514] Processing DeleteTablet for tablet 1602129ae4d042048001a370ffeb4d92 with delete_type TABLET_DATA_TOMBSTONED (TS 9a340c22b2014883b6bf4e891be77b34 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52084
I20240503 07:58:52.871079 6006 tablet_replica.cc:330] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: stopping tablet replica
I20240503 07:58:52.871197 6006 raft_consensus.cc:2229] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:58:52.871294 6006 raft_consensus.cc:2258] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:58:52.872399 6006 ts_tablet_manager.cc:1875] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:58:52.874121 6006 ts_tablet_manager.cc:1888] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:58:52.874186 6006 log.cc:1196] T 1602129ae4d042048001a370ffeb4d92 P 9a340c22b2014883b6bf4e891be77b34: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1602129ae4d042048001a370ffeb4d92
I20240503 07:58:52.874596 2609 catalog_manager.cc:4830] TS 9a340c22b2014883b6bf4e891be77b34 (127.0.99.129:41775): tablet 1602129ae4d042048001a370ffeb4d92 (table rebalance_test_table_3 [id=460d76968630465590265d7192efbf8f]) successfully deleted
I20240503 07:58:52.995759 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:52.996212 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.002282 2745 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.006003 5596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.010725 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.016922 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.019438 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.025828 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.100690 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.107475 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.108042 2745 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.111363 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.111958 5596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.117545 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.124572 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.129025 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.221377 2745 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.221839 4279 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.227317 3876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.228238 4410 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.228258 5596 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.229874 2876 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.241240 4007 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:58:53.259698 4139 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
152cf5a0fb8c45389266683debe2a67e | 127.0.99.190:34925 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43981 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
377b7db17e524c0385a4cfbdd4803cf5 | 127.0.99.134:33903 | HEALTHY | <none> | 2 | 0
5294114bec9e421a866b4f999d3bb6bf | 127.0.99.131:40707 | HEALTHY | <none> | 2 | 0
794c7a3190fe4b0fafdde8f50af31b86 | 127.0.99.136:43801 | HEALTHY | <none> | 2 | 0
9a340c22b2014883b6bf4e891be77b34 | 127.0.99.129:41775 | HEALTHY | <none> | 2 | 0
afda07f6bc6e420d81525f220d2441ce | 127.0.99.135:46165 | HEALTHY | <none> | 2 | 0
b4316a9fd1e64780a693b466700774bd | 127.0.99.133:34145 | HEALTHY | <none> | 2 | 0
c66f48bb18804160915381eac766f6fe | 127.0.99.130:39115 | HEALTHY | <none> | 2 | 0
f9266aaeb8404667ae633cb22af9d372 | 127.0.99.132:43805 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:41775
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:39115
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:40707
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43805
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:34145
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:33903
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:46165
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:43801
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:41775
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:39115
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:40707
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43805
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:34145
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:33903
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:46165
server_dump_info_path | /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:43801
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:43981 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 07:58:53.279608 398 log_verifier.cc:126] Checking tablet 0d27df72acf7499eb7b57aded052fc97
I20240503 07:58:53.285557 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0d27df72acf7499eb7b57aded052fc97
I20240503 07:58:53.285604 398 log_verifier.cc:126] Checking tablet 1602129ae4d042048001a370ffeb4d92
I20240503 07:58:53.293820 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1602129ae4d042048001a370ffeb4d92
I20240503 07:58:53.293875 398 log_verifier.cc:126] Checking tablet 18b0318923a944ae880627a8a0c7f7ab
I20240503 07:58:53.299299 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 18b0318923a944ae880627a8a0c7f7ab
I20240503 07:58:53.299353 398 log_verifier.cc:126] Checking tablet 2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:53.304981 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ea6ab00e8f44f0888ecdb9a3084aac4
I20240503 07:58:53.305023 398 log_verifier.cc:126] Checking tablet 5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:53.310436 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5f714d0cb8cf4288ab1d64b851ea7e57
I20240503 07:58:53.310482 398 log_verifier.cc:126] Checking tablet 67633b2413ef4567bf371391a07772af
I20240503 07:58:53.315979 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67633b2413ef4567bf371391a07772af
I20240503 07:58:53.316020 398 log_verifier.cc:126] Checking tablet 79bc252e1a9c40778cb8c72ff8b8e662
I20240503 07:58:53.321312 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 79bc252e1a9c40778cb8c72ff8b8e662
I20240503 07:58:53.321360 398 log_verifier.cc:126] Checking tablet 85ac898d1f5344b8bb6dc00f9db27ae3
I20240503 07:58:53.325649 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 85ac898d1f5344b8bb6dc00f9db27ae3
I20240503 07:58:53.325690 398 log_verifier.cc:126] Checking tablet 974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:53.333464 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 974d562e7c66487a8f767b3482a2daa6
I20240503 07:58:53.333514 398 log_verifier.cc:126] Checking tablet 9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:53.339035 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9d66454fdac8462ab56d5b23d4b80780
I20240503 07:58:53.339079 398 log_verifier.cc:126] Checking tablet a146e960f86f4876881041ce1f33bafe
I20240503 07:58:53.344647 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet a146e960f86f4876881041ce1f33bafe
I20240503 07:58:53.344689 398 log_verifier.cc:126] Checking tablet aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:53.350179 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet aaba11f99cc843d2b67cdc46fa666350
I20240503 07:58:53.350221 398 log_verifier.cc:126] Checking tablet b705940a75a541d9816297f580ed5b57
I20240503 07:58:53.355728 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b705940a75a541d9816297f580ed5b57
I20240503 07:58:53.355770 398 log_verifier.cc:126] Checking tablet bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:53.361449 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bd25b211a6b54c1fa079b799a8e6af9d
I20240503 07:58:53.361490 398 log_verifier.cc:126] Checking tablet ceab64ea8ffb4e33be8bc5c851b6119c
I20240503 07:58:53.366676 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ceab64ea8ffb4e33be8bc5c851b6119c
I20240503 07:58:53.366719 398 log_verifier.cc:126] Checking tablet d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:53.372262 398 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d8b7708dcbaf44178da851db2e62d702
I20240503 07:58:53.372736 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2683
I20240503 07:58:53.380513 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2814
I20240503 07:58:53.386389 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3814
I20240503 07:58:53.391139 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 3945
I20240503 07:58:53.395388 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 4076
I20240503 07:58:53.399539 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 4209
I20240503 07:58:53.403735 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 4348
I20240503 07:58:53.408108 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 5534
I20240503 07:58:53.412266 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 2592
2024-05-03T07:58:53Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (15958 ms)
[----------] 1 test from RebalanceParamTest (15958 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 07:58:53.444491 398 test_util.cc:268] Using random seed: 390189448
I20240503 07:58:53.445330 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:58:53.445381 398 ts_itest-base.cc:113] --------------
I20240503 07:58:53.445397 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:58:53.445411 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:58:53.445438 398 ts_itest-base.cc:116] --------------
2024-05-03T07:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:58:53Z Disabled control of system clock
I20240503 07:58:53.453181 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39595
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:39595
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:58:53.553042 6126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.553231 6126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.553269 6126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.556098 6126 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:53.556156 6126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:58:53.556181 6126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.556207 6126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:58:53.556232 6126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:58:53.559664 6126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:39595
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:39595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.560343 6126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.561295 6126 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.568150 6131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.568921 6132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.569451 6134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.570133 6126 server_base.cc:1034] running on GCE node
I20240503 07:58:53.570538 6126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.571293 6126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.572439 6126 hybrid_clock.cc:648] HybridClock initialized: now 1714723133572429 us; error 23 us; skew 500 ppm
I20240503 07:58:53.574136 6126 webserver.cc:460] Webserver started at http://127.0.99.190:39031/ using document root <none> and password file <none>
I20240503 07:58:53.574628 6126 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.574678 6126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.574826 6126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.576237 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e0b9972b770f4d3bb0f76d2fbe669505"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.576673 6126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e0b9972b770f4d3bb0f76d2fbe669505"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.579528 6126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240503 07:58:53.581436 6140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.582271 6126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:53.582374 6126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e0b9972b770f4d3bb0f76d2fbe669505"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.582444 6126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.598727 6126 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.599203 6126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.599331 6126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.605019 6126 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:39595
I20240503 07:58:53.605032 6192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:39595 every 8 connection(s)
I20240503 07:58:53.605841 6126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:58:53.607147 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6126
I20240503 07:58:53.607251 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:58:53.608183 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:53.613741 6193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Bootstrap starting.
I20240503 07:58:53.615674 6193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:53.616319 6193 log.cc:824] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:53.617686 6193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: No bootstrap required, opened a new log
I20240503 07:58:53.619897 6193 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } }
I20240503 07:58:53.620055 6193 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:53.620110 6193 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b9972b770f4d3bb0f76d2fbe669505, State: Initialized, Role: FOLLOWER
I20240503 07:58:53.620477 6193 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } }
I20240503 07:58:53.620596 6193 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:58:53.620649 6193 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:58:53.620738 6193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:53.621353 6193 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } }
I20240503 07:58:53.621627 6193 leader_election.cc:302] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b9972b770f4d3bb0f76d2fbe669505; no voters:
I20240503 07:58:53.621858 6193 leader_election.cc:288] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:58:53.622129 6198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:53.622367 6198 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [term 1 LEADER]: Becoming Leader. State: Replica: e0b9972b770f4d3bb0f76d2fbe669505, State: Running, Role: LEADER
I20240503 07:58:53.622632 6198 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } }
I20240503 07:58:53.622802 6193 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:58:53.623921 6200 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b9972b770f4d3bb0f76d2fbe669505. Latest consensus state: current_term: 1 leader_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } } }
I20240503 07:58:53.623951 6199 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b9972b770f4d3bb0f76d2fbe669505" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 39595 } } }
I20240503 07:58:53.624024 6200 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:53.624037 6199 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505 [sys.catalog]: This master's current role is: LEADER
I20240503 07:58:53.624421 6208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:58:53.627053 6208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:58:53.631875 6208 catalog_manager.cc:1347] Generated new cluster ID: 808453908e82453c85dd4ab3aec487d6
I20240503 07:58:53.631934 6208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:58:53.641294 6208 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:58:53.642186 6208 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:58:53.655834 6208 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Generated new TSK 0
I20240503 07:58:53.656430 6208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:58:53.660095 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:53.758436 6217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.758630 6217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.758687 6217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.761631 6217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:53.761691 6217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.761772 6217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:58:53.765345 6217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.766094 6217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.767093 6217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.775291 6223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.776062 6222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.776613 6225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.777331 6217 server_base.cc:1034] running on GCE node
I20240503 07:58:53.777688 6217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.778174 6217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.779306 6217 hybrid_clock.cc:648] HybridClock initialized: now 1714723133779275 us; error 47 us; skew 500 ppm
I20240503 07:58:53.781013 6217 webserver.cc:460] Webserver started at http://127.0.99.129:44905/ using document root <none> and password file <none>
I20240503 07:58:53.781522 6217 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.781574 6217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.781724 6217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.783218 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c167a0a433d34b74affed13cbf96e302"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.783643 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c167a0a433d34b74affed13cbf96e302"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.786691 6217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:58:53.788517 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.789414 6217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:53.789518 6217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c167a0a433d34b74affed13cbf96e302"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.789592 6217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.802081 6217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.802529 6217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.802651 6217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.803079 6217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.803825 6217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.803870 6217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.803903 6217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.803916 6217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.811163 6217 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37267
I20240503 07:58:53.811209 6344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37267 every 8 connection(s)
I20240503 07:58:53.811918 6217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:58:53.814774 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6217
I20240503 07:58:53.814857 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:58:53.816830 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:53.819958 6345 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:53.820137 6345 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.820566 6345 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:53.821985 6157 ts_manager.cc:194] Registered new tserver with Master: c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:58:53.823513 6157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:59079
W20240503 07:58:53.912604 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:53.912765 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:53.912822 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:53.915622 6348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:53.915673 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:53.915735 6348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:58:53.919056 6348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:53.919696 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:53.920620 6348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:53.928504 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.929257 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:53.929908 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:53.930528 6348 server_base.cc:1034] running on GCE node
I20240503 07:58:53.930871 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:53.931315 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:53.932466 6348 hybrid_clock.cc:648] HybridClock initialized: now 1714723133932449 us; error 29 us; skew 500 ppm
I20240503 07:58:53.934170 6348 webserver.cc:460] Webserver started at http://127.0.99.130:46653/ using document root <none> and password file <none>
I20240503 07:58:53.934638 6348 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:53.934690 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:53.934847 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:53.936305 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.936722 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.939767 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240503 07:58:53.941771 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.942648 6348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:53.942741 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
format_stamp: "Formatted at 2024-05-03 07:58:53 on dist-test-slave-7tt0"
I20240503 07:58:53.942831 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:53.977212 6348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:53.977705 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:53.977849 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:53.978310 6348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:53.979115 6348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:53.979164 6348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.979207 6348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:53.979239 6348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:53.987717 6348 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:41651
I20240503 07:58:53.987736 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:41651 every 8 connection(s)
I20240503 07:58:53.988544 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:58:53.991670 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6348
I20240503 07:58:53.991784 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:58:53.993906 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:53.997020 6476 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:53.997211 6476 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:53.997681 6476 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:53.998603 6157 ts_manager.cc:194] Registered new tserver with Master: d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:58:53.999303 6157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:55113
W20240503 07:58:54.093133 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.093318 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.093365 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.096232 6479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.096288 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.096355 6479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:58:54.099733 6479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.100442 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.101447 6479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.109463 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.109886 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.110716 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.111398 6479 server_base.cc:1034] running on GCE node
I20240503 07:58:54.111732 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.112232 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.113391 6479 hybrid_clock.cc:648] HybridClock initialized: now 1714723134113379 us; error 28 us; skew 500 ppm
I20240503 07:58:54.115079 6479 webserver.cc:460] Webserver started at http://127.0.99.131:45237/ using document root <none> and password file <none>
I20240503 07:58:54.115628 6479 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.115691 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.115867 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.117460 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ad910f92ef9d4db8a8962cecba289f8a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.117920 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ad910f92ef9d4db8a8962cecba289f8a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.121094 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:54.123171 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.124151 6479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.124265 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ad910f92ef9d4db8a8962cecba289f8a"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.124382 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.135627 6479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.136068 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.136204 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.136636 6479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.137432 6479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.137476 6479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.137523 6479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.137549 6479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.145340 6479 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45693
I20240503 07:58:54.145370 6606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45693 every 8 connection(s)
I20240503 07:58:54.146150 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:58:54.148952 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6479
I20240503 07:58:54.149047 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:58:54.150966 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:54.154764 6607 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.154959 6607 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.155397 6607 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.156381 6157 ts_manager.cc:194] Registered new tserver with Master: ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:54.157130 6157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:50185
W20240503 07:58:54.249347 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.249527 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.249567 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.253578 6610 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.253644 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.253728 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:58:54.257432 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.258118 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.259052 6610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.266829 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.267264 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.268059 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.268707 6610 server_base.cc:1034] running on GCE node
I20240503 07:58:54.269145 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.269606 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.270752 6610 hybrid_clock.cc:648] HybridClock initialized: now 1714723134270731 us; error 36 us; skew 500 ppm
I20240503 07:58:54.272326 6610 webserver.cc:460] Webserver started at http://127.0.99.132:33085/ using document root <none> and password file <none>
I20240503 07:58:54.272830 6610 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.272886 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.273043 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.274446 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a97073f89fd44702bccb74a9b76f858d"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.274842 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a97073f89fd44702bccb74a9b76f858d"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.277748 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 07:58:54.279806 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.280699 6610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.280824 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a97073f89fd44702bccb74a9b76f858d"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.280906 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.295198 6610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.295683 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.295814 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.296253 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.297083 6610 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.297127 6610 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.297173 6610 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.297200 6610 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.304479 6610 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39733
I20240503 07:58:54.304508 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39733 every 8 connection(s)
I20240503 07:58:54.305332 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:58:54.305721 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6610
I20240503 07:58:54.305822 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:58:54.307849 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:54.314014 6738 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.314196 6738 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.314639 6738 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.316407 6157 ts_manager.cc:194] Registered new tserver with Master: a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:58:54.316943 6157 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:60983
W20240503 07:58:54.406062 6741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.406241 6741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.406298 6741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.409153 6741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.409207 6741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.409287 6741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:58:54.412592 6741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.413311 6741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.414249 6741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.421972 6749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.422693 6746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.423231 6747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.423992 6741 server_base.cc:1034] running on GCE node
I20240503 07:58:54.424322 6741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.424824 6741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.425972 6741 hybrid_clock.cc:648] HybridClock initialized: now 1714723134425953 us; error 36 us; skew 500 ppm
I20240503 07:58:54.427709 6741 webserver.cc:460] Webserver started at http://127.0.99.133:35381/ using document root <none> and password file <none>
I20240503 07:58:54.428193 6741 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.428252 6741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.428436 6741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.429949 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c8d71787690245e681c85a087086e7ed"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.430379 6741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c8d71787690245e681c85a087086e7ed"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.433491 6741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 07:58:54.435537 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.436507 6741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.436607 6741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c8d71787690245e681c85a087086e7ed"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.436684 6741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.454552 6741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.455044 6741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.455178 6741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.455649 6741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.456416 6741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.456461 6741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.456507 6741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.456535 6741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.464028 6741 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39949
I20240503 07:58:54.464077 6868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39949 every 8 connection(s)
I20240503 07:58:54.464836 6741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:58:54.472390 6869 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.472571 6869 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.472851 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6741
I20240503 07:58:54.472944 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:58:54.473075 6869 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.473912 6156 ts_manager.cc:194] Registered new tserver with Master: c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:58:54.474465 6156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:41575
I20240503 07:58:54.474771 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:58:54.573431 6872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.573601 6872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.573640 6872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.576539 6872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.576588 6872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.576647 6872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:58:54.580029 6872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.6872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.580715 6872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.581717 6872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.589576 6877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.590214 6880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.590981 6878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.591673 6872 server_base.cc:1034] running on GCE node
I20240503 07:58:54.592044 6872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.592607 6872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.593772 6872 hybrid_clock.cc:648] HybridClock initialized: now 1714723134593755 us; error 39 us; skew 500 ppm
I20240503 07:58:54.595407 6872 webserver.cc:460] Webserver started at http://127.0.99.134:44279/ using document root <none> and password file <none>
I20240503 07:58:54.595896 6872 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.595949 6872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.596099 6872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.597718 6872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.598194 6872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.601401 6872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:58:54.603421 6886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.604404 6872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.604514 6872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.604593 6872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.632598 6872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.633133 6872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.633268 6872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.633729 6872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.634518 6872 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.634563 6872 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.634610 6872 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.634637 6872 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.642297 6872 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40925
I20240503 07:58:54.642325 6999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40925 every 8 connection(s)
I20240503 07:58:54.643090 6872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:58:54.649510 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 6872
I20240503 07:58:54.649616 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:58:54.651321 7000 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.651504 7000 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.651687 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:54.651973 7000 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.652873 6156 ts_manager.cc:194] Registered new tserver with Master: 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:58:54.653528 6156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:47569
W20240503 07:58:54.750737 7003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.750926 7003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.750993 7003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.753897 7003 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.753962 7003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.754047 7003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:58:54.757432 7003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.7003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.758162 7003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.759095 7003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.766764 7011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.767552 7008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.768121 7009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.768800 7003 server_base.cc:1034] running on GCE node
I20240503 07:58:54.769150 7003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.769618 7003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.770769 7003 hybrid_clock.cc:648] HybridClock initialized: now 1714723134770761 us; error 28 us; skew 500 ppm
I20240503 07:58:54.772411 7003 webserver.cc:460] Webserver started at http://127.0.99.135:33603/ using document root <none> and password file <none>
I20240503 07:58:54.772905 7003 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.772961 7003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.773125 7003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.774545 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "616fdda6ba714cb7be01a564de2b9db2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.774945 7003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "616fdda6ba714cb7be01a564de2b9db2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.777832 7003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 07:58:54.779703 7017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.780570 7003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.780666 7003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "616fdda6ba714cb7be01a564de2b9db2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.780733 7003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.801023 7003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.801479 7003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.801622 7003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.802065 7003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.802845 7003 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.802894 7003 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.802945 7003 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.802978 7003 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.811297 7003 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39417
I20240503 07:58:54.811342 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39417 every 8 connection(s)
I20240503 07:58:54.812294 7003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:58:54.816622 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 7003
I20240503 07:58:54.816725 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:58:54.819180 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:58:54.820822 7131 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.821028 7131 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.821509 7131 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.822422 6156 ts_manager.cc:194] Registered new tserver with Master: 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:58:54.823114 6156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:47819
I20240503 07:58:54.825527 6345 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
W20240503 07:58:54.918820 7134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:54.919005 7134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:58:54.919067 7134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:54.921944 7134 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:58:54.922008 7134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:58:54.922094 7134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:58:54.925542 7134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.7134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:58:54.926249 7134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:58:54.927210 7134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:58:54.935137 7142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.935772 7139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:58:54.936375 7140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:58:54.937047 7134 server_base.cc:1034] running on GCE node
I20240503 07:58:54.937366 7134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:58:54.937822 7134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:58:54.938963 7134 hybrid_clock.cc:648] HybridClock initialized: now 1714723134938953 us; error 33 us; skew 500 ppm
I20240503 07:58:54.940618 7134 webserver.cc:460] Webserver started at http://127.0.99.136:41261/ using document root <none> and password file <none>
I20240503 07:58:54.941135 7134 fs_manager.cc:359] Metadata directory not provided
I20240503 07:58:54.941193 7134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:58:54.941358 7134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:58:54.942781 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "3d08a06d53b14381bee6223c4e1a6027"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.943192 7134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "3d08a06d53b14381bee6223c4e1a6027"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.948928 7134 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20240503 07:58:54.950949 7148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.951862 7134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:58:54.951962 7134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "3d08a06d53b14381bee6223c4e1a6027"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:58:54.952044 7134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:58:54.964283 7134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:58:54.964749 7134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:58:54.964948 7134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:58:54.965404 7134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:58:54.966158 7134 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:58:54.966202 7134 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.966246 7134 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:58:54.966272 7134 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:58:54.973977 7134 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41089
I20240503 07:58:54.974009 7261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41089 every 8 connection(s)
I20240503 07:58:54.974720 7134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:58:54.975083 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 7134
I20240503 07:58:54.975188 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:58:54.982468 7262 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:58:54.982630 7262 heartbeater.cc:464] Registering TS with master...
I20240503 07:58:54.983053 7262 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:58:54.983912 6156 ts_manager.cc:194] Registered new tserver with Master: 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:58:54.984416 6156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:43421
I20240503 07:58:54.989068 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:54.997942 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:58:54.998001 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6741
I20240503 07:58:55.001127 6476 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:58:55.003193 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6872
I20240503 07:58:55.007877 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 7003
I20240503 07:58:55.011894 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 7134
I20240503 07:58:55.158885 6607 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:58:55.318606 6738 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:58:58.767424 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:58.786549 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd030945264526a0570de935618ff5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.786554 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd030945264526a0570de935618ff5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.786698 6410 tablet_service.cc:1467] Processing CreateTablet for tablet f5b2f846e84c4dbc9fab564845def15d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.786554 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.787479 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b2f846e84c4dbc9fab564845def15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.787479 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd030945264526a0570de935618ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.787482 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b9249ed9a94cb2a91bb79ba5d077b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.787760 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd030945264526a0570de935618ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.789103 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.789328 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b9249ed9a94cb2a91bb79ba5d077b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.791042 6540 tablet_service.cc:1467] Processing CreateTablet for tablet f5b2f846e84c4dbc9fab564845def15d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.791265 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b2f846e84c4dbc9fab564845def15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.793798 7283 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:58:58.795250 7282 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:58:58.795858 7283 tablet_bootstrap.cc:654] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.796638 7283 log.cc:824] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.796666 6279 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd030945264526a0570de935618ff5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.797108 7282 tablet_bootstrap.cc:654] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.797219 6278 tablet_service.cc:1467] Processing CreateTablet for tablet f5b2f846e84c4dbc9fab564845def15d (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.797523 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd030945264526a0570de935618ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.796666 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:58.797770 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b2f846e84c4dbc9fab564845def15d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.797828 7282 log.cc:824] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.798326 7283 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:58:58.798473 7283 ts_tablet_manager.cc:1367] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:58:58.799263 7282 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:58:58.799413 7282 ts_tablet_manager.cc:1367] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:58:58.800799 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b9249ed9a94cb2a91bb79ba5d077b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:58.801823 7283 raft_consensus.cc:348] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.802014 7283 raft_consensus.cc:374] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.802062 7283 raft_consensus.cc:729] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.802652 7283 consensus_queue.cc:260] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.802795 7282 raft_consensus.cc:348] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.802965 7282 raft_consensus.cc:374] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.803020 7282 raft_consensus.cc:729] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.803400 7282 consensus_queue.cc:260] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.803719 7283 ts_tablet_manager.cc:1398] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 07:58:58.804157 7282 ts_tablet_manager.cc:1398] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:58:58.804409 7286 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:58:58.804458 7282 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:58:58.805105 7283 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:58:58.805267 7282 tablet_bootstrap.cc:654] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.805912 7283 tablet_bootstrap.cc:654] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.806198 7282 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:58:58.806262 7282 ts_tablet_manager.cc:1367] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:58.806262 7286 tablet_bootstrap.cc:654] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.806737 7282 raft_consensus.cc:348] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.806850 7282 raft_consensus.cc:374] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.806854 7283 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:58:58.806885 7282 raft_consensus.cc:729] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.806926 7283 ts_tablet_manager.cc:1367] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:58.806944 7286 log.cc:824] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.806985 7282 consensus_queue.cc:260] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.807289 7282 ts_tablet_manager.cc:1398] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:58.807381 7283 raft_consensus.cc:348] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.807490 7283 raft_consensus.cc:374] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.807521 7283 raft_consensus.cc:729] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.807623 7283 consensus_queue.cc:260] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.807902 7283 ts_tablet_manager.cc:1398] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.807952 7287 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:58:58.808800 7286 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:58:58.808940 7286 ts_tablet_manager.cc:1367] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:58.809459 7287 tablet_bootstrap.cc:654] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.810441 7287 log.cc:824] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Log is configured to *not* fsync() on all Append() calls
I20240503 07:58:58.811607 7287 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:58:58.811749 7287 ts_tablet_manager.cc:1367] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:58:58.812181 7286 raft_consensus.cc:348] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.812371 7286 raft_consensus.cc:374] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.812425 7286 raft_consensus.cc:729] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.812803 7286 consensus_queue.cc:260] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.813534 7286 ts_tablet_manager.cc:1398] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:58:58.813732 7286 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:58:58.814131 7287 raft_consensus.cc:348] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.814273 7287 raft_consensus.cc:374] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.814321 7287 raft_consensus.cc:729] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.814509 7286 tablet_bootstrap.cc:654] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.814692 7287 consensus_queue.cc:260] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.815424 7287 ts_tablet_manager.cc:1398] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 07:58:58.815639 7287 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:58:58.815883 7286 tablet_bootstrap.cc:492] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:58:58.816006 7286 ts_tablet_manager.cc:1367] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
W20240503 07:58:58.816208 6346 tablet.cc:2369] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:58.816434 7287 tablet_bootstrap.cc:654] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.816628 7286 raft_consensus.cc:348] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.816810 7286 raft_consensus.cc:374] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.816891 7286 raft_consensus.cc:729] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.817077 7286 consensus_queue.cc:260] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.817423 7287 tablet_bootstrap.cc:492] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:58:58.817452 7286 ts_tablet_manager.cc:1398] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.817504 7287 ts_tablet_manager.cc:1367] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:58.817925 7287 raft_consensus.cc:348] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.818001 7287 raft_consensus.cc:374] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.818029 7287 raft_consensus.cc:729] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.818137 7287 consensus_queue.cc:260] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.818351 7287 ts_tablet_manager.cc:1398] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:58.818480 7287 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:58:58.819244 7287 tablet_bootstrap.cc:654] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:58.820349 7287 tablet_bootstrap.cc:492] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:58:58.820430 7287 ts_tablet_manager.cc:1367] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:58.820748 7287 raft_consensus.cc:348] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.820900 7287 raft_consensus.cc:374] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:58.820932 7287 raft_consensus.cc:729] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:58:58.821046 7287 consensus_queue.cc:260] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:58.821247 7287 ts_tablet_manager.cc:1398] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:58:58.899962 6608 tablet.cc:2369] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:58.914844 7296 raft_consensus.cc:482] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.915019 7296 raft_consensus.cc:504] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.916043 7289 raft_consensus.cc:482] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:58.916100 7296 leader_election.cc:288] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:58:58.916213 7289 raft_consensus.cc:504] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.917444 7289 leader_election.cc:288] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:58:58.920977 6692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:58:58.921051 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:58:58.921236 6430 raft_consensus.cc:2454] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:58:58.921253 6692 raft_consensus.cc:2454] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:58:58.921672 6235 leader_election.cc:302] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:58:58.921927 7296 raft_consensus.cc:2781] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.921989 7296 raft_consensus.cc:482] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.922030 7296 raft_consensus.cc:3037] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.922312 6692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:58:58.922448 6692 raft_consensus.cc:2454] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:58:58.922736 6365 leader_election.cc:302] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:58:58.922976 7289 raft_consensus.cc:2781] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:58.923051 7289 raft_consensus.cc:482] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:58.923101 7289 raft_consensus.cc:3037] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.923121 7296 raft_consensus.cc:504] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.923283 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:58:58.923455 7296 leader_election.cc:288] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:58:58.923499 6299 raft_consensus.cc:2379] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1f36d57661b4429a4be8d10b58d8ff7 in current term 1: Already voted for candidate c167a0a433d34b74affed13cbf96e302 in this term.
I20240503 07:58:58.923727 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:58:58.923967 6692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:58:58.924062 6692 raft_consensus.cc:3037] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:58.924203 7289 raft_consensus.cc:504] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.924571 7289 leader_election.cc:288] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:58:58.924760 6430 raft_consensus.cc:2379] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c167a0a433d34b74affed13cbf96e302 in current term 1: Already voted for candidate d1f36d57661b4429a4be8d10b58d8ff7 in this term.
I20240503 07:58:58.924908 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:58:58.925026 6692 raft_consensus.cc:2454] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:58:58.925026 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b9249ed9a94cb2a91bb79ba5d077b2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:58:58.925153 6299 raft_consensus.cc:2379] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1f36d57661b4429a4be8d10b58d8ff7 in current term 1: Already voted for candidate c167a0a433d34b74affed13cbf96e302 in this term.
I20240503 07:58:58.925431 6234 leader_election.cc:302] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:58:58.925475 6365 leader_election.cc:302] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1f36d57661b4429a4be8d10b58d8ff7; no voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302
I20240503 07:58:58.925581 7296 raft_consensus.cc:2781] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:58.925712 7289 raft_consensus.cc:2726] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 07:58:58.925765 7296 raft_consensus.cc:686] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:58:58.926017 7296 consensus_queue.cc:237] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:58.928728 6157 catalog_manager.cc:5484] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:58:58.992468 6477 tablet.cc:2369] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:58:59.059746 6739 tablet.cc:2369] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:58:59.161247 7296 raft_consensus.cc:482] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.161361 7296 raft_consensus.cc:504] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.161785 7296 leader_election.cc:288] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.162179 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b2f846e84c4dbc9fab564845def15d" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:58:59.162360 6430 raft_consensus.cc:2454] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:58:59.162653 6235 leader_election.cc:302] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:58:59.162811 7296 raft_consensus.cc:2781] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.162889 7296 raft_consensus.cc:482] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.162926 7296 raft_consensus.cc:3037] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.163856 7296 raft_consensus.cc:504] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.164165 7296 leader_election.cc:288] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.164496 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b2f846e84c4dbc9fab564845def15d" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:58:59.164619 6430 raft_consensus.cc:3037] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.165470 6430 raft_consensus.cc:2454] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:58:59.165410 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b2f846e84c4dbc9fab564845def15d" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:58:59.165412 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b2f846e84c4dbc9fab564845def15d" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:58:59.165642 6560 raft_consensus.cc:3037] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.166586 6560 raft_consensus.cc:2454] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:58:59.166852 6232 leader_election.cc:302] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:58:59.167014 7296 raft_consensus.cc:2781] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.167107 7296 raft_consensus.cc:686] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:58:59.167238 7296 consensus_queue.cc:237] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.168695 6157 catalog_manager.cc:5484] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.234232 7295 raft_consensus.cc:482] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.234346 7295 raft_consensus.cc:504] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:59.235298 7295 leader_election.cc:288] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:58:59.239063 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd030945264526a0570de935618ff5" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:58:59.239063 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd030945264526a0570de935618ff5" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:58:59.239195 6560 raft_consensus.cc:2454] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:58:59.239210 6299 raft_consensus.cc:2454] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:58:59.239521 6625 leader_election.cc:302] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:58:59.239734 7295 raft_consensus.cc:2781] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.239797 7295 raft_consensus.cc:482] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.239840 7295 raft_consensus.cc:3037] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.240600 7295 raft_consensus.cc:504] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:59.240926 7295 leader_election.cc:288] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:58:59.241202 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd030945264526a0570de935618ff5" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:58:59.241263 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd030945264526a0570de935618ff5" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:58:59.241298 6560 raft_consensus.cc:3037] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.241370 6299 raft_consensus.cc:3037] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.241986 6560 raft_consensus.cc:2454] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:58:59.242085 6299 raft_consensus.cc:2454] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:58:59.242225 6625 leader_election.cc:302] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:58:59.242381 7295 raft_consensus.cc:2781] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.242564 7295 raft_consensus.cc:686] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:58:59.242856 7295 consensus_queue.cc:237] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:58:59.245658 6157 catalog_manager.cc:5484] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.418666 7296 consensus_queue.cc:1035] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.423513 7315 consensus_queue.cc:1035] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:58:59.443307 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:58:59.443507 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:58:59.446507 7312 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:58:59.542148 6299 raft_consensus.cc:1261] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.542852 6560 raft_consensus.cc:1261] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.543691 7295 consensus_queue.cc:1035] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.544135 7295 consensus_queue.cc:1035] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.550166 6429 raft_consensus.cc:1261] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.551992 7315 consensus_queue.cc:1035] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.552455 6560 raft_consensus.cc:1261] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:58:59.552959 7315 consensus_queue.cc:1035] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:58:59.560191 7318 mvcc.cc:205] Tried to move back new op lower bound from 7023505979571875840 to 7023505978340986880. Current Snapshot: MvccSnapshot[applied={T|T < 7023505979555008512}]
I20240503 07:58:59.562321 7314 mvcc.cc:205] Tried to move back new op lower bound from 7023505979589980160 to 7023505978030137344. Current Snapshot: MvccSnapshot[applied={T|T < 7023505979568877568}]
I20240503 07:58:59.569497 7350 mvcc.cc:205] Tried to move back new op lower bound from 7023505979555008512 to 7023505978340986880. Current Snapshot: MvccSnapshot[applied={T|T < 7023505979555008512}]
I20240503 07:58:59.577013 7316 mvcc.cc:205] Tried to move back new op lower bound from 7023505979589980160 to 7023505978030137344. Current Snapshot: MvccSnapshot[applied={T|T < 7023505979568877568}]
INSERT report
rows total: 2000
time total: 121.79 ms
time per row: 0.0608951 ms
I20240503 07:58:59.608698 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:58:59.613606 6540 tablet_service.cc:1467] Processing CreateTablet for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.613691 6409 tablet_service.cc:1467] Processing CreateTablet for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.613885 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.613937 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.614452 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.614672 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a63d4f2c744129a47ba3c4b7ef4f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.616461 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 2f943ecfe59f432c861a54b16c62572f (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.616560 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 2f943ecfe59f432c861a54b16c62572f (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.616561 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 2f943ecfe59f432c861a54b16c62572f (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.616662 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f943ecfe59f432c861a54b16c62572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.616763 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f943ecfe59f432c861a54b16c62572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.616802 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f943ecfe59f432c861a54b16c62572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.616832 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 435f73fb9f384b10b353df19037bcac3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.617031 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f73fb9f384b10b353df19037bcac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.618676 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 435f73fb9f384b10b353df19037bcac3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.618739 7377 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:58:59.618871 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f73fb9f384b10b353df19037bcac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.619177 7374 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:58:59.619387 7375 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:58:59.619495 7377 tablet_bootstrap.cc:654] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.620028 7376 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:58:59.620329 7374 tablet_bootstrap.cc:654] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.620870 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 435f73fb9f384b10b353df19037bcac3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:58:59.620925 7376 tablet_bootstrap.cc:654] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.621073 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f73fb9f384b10b353df19037bcac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:58:59.621516 7375 tablet_bootstrap.cc:654] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.622098 7377 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:58:59.622186 7377 ts_tablet_manager.cc:1367] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:58:59.622483 7374 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:58:59.622563 7374 ts_tablet_manager.cc:1367] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:58:59.622560 7377 raft_consensus.cc:348] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.622663 7377 raft_consensus.cc:374] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.622715 7377 raft_consensus.cc:729] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.622848 7377 consensus_queue.cc:260] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.622949 7374 raft_consensus.cc:348] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.623049 7374 raft_consensus.cc:374] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.623082 7374 raft_consensus.cc:729] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.623121 7377 ts_tablet_manager.cc:1398] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.623198 7374 consensus_queue.cc:260] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.623601 7375 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:58:59.623675 7375 ts_tablet_manager.cc:1367] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:58:59.623724 7376 tablet_bootstrap.cc:492] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:58:59.623824 7376 ts_tablet_manager.cc:1367] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:58:59.624094 7375 raft_consensus.cc:348] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.624184 7375 raft_consensus.cc:374] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.624213 7375 raft_consensus.cc:729] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.624231 7376 raft_consensus.cc:348] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.624320 7376 raft_consensus.cc:374] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.624349 7376 raft_consensus.cc:729] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.624321 7375 consensus_queue.cc:260] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.624452 7376 consensus_queue.cc:260] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.624619 7374 ts_tablet_manager.cc:1398] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:59.624887 7375 ts_tablet_manager.cc:1398] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.625084 7375 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:58:59.625869 7374 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:58:59.625878 7375 tablet_bootstrap.cc:654] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.626782 7375 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:58:59.626843 7375 ts_tablet_manager.cc:1367] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:59.627135 7376 ts_tablet_manager.cc:1398] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:58:59.627197 7374 tablet_bootstrap.cc:654] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.627226 7375 raft_consensus.cc:348] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.627326 7375 raft_consensus.cc:374] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.627360 7376 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:58:59.627357 7375 raft_consensus.cc:729] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.627570 7375 consensus_queue.cc:260] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.627851 7375 ts_tablet_manager.cc:1398] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.627993 7375 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:58:59.628338 7376 tablet_bootstrap.cc:654] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.628793 7375 tablet_bootstrap.cc:654] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.629240 7376 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:58:59.629312 7376 ts_tablet_manager.cc:1367] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:58:59.629694 7376 raft_consensus.cc:348] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.629797 7376 raft_consensus.cc:374] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.629828 7376 raft_consensus.cc:729] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.629938 7376 consensus_queue.cc:260] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.630141 7376 ts_tablet_manager.cc:1398] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:59.630190 7375 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:58:59.630275 7375 ts_tablet_manager.cc:1367] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:58:59.630662 7375 raft_consensus.cc:348] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.630755 7375 raft_consensus.cc:374] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.630786 7375 raft_consensus.cc:729] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.630918 7375 consensus_queue.cc:260] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.631215 7375 ts_tablet_manager.cc:1398] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:58:59.632083 7374 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:58:59.632179 7374 ts_tablet_manager.cc:1367] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:58:59.632529 7374 raft_consensus.cc:348] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.632616 7374 raft_consensus.cc:374] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.632645 7374 raft_consensus.cc:729] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.632755 7374 consensus_queue.cc:260] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.633004 7374 ts_tablet_manager.cc:1398] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:58:59.633144 7374 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:58:59.633949 7374 tablet_bootstrap.cc:654] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:58:59.634771 7374 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:58:59.634840 7374 ts_tablet_manager.cc:1367] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:58:59.635155 7374 raft_consensus.cc:348] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.635228 7374 raft_consensus.cc:374] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:58:59.635255 7374 raft_consensus.cc:729] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:58:59.635345 7374 consensus_queue.cc:260] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.635558 7374 ts_tablet_manager.cc:1398] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:58:59.803573 7315 raft_consensus.cc:482] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.803700 7315 raft_consensus.cc:504] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.804105 7315 leader_election.cc:288] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.804517 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f943ecfe59f432c861a54b16c62572f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:58:59.804589 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f943ecfe59f432c861a54b16c62572f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:58:59.804683 6561 raft_consensus.cc:2454] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:58:59.804716 6429 raft_consensus.cc:2454] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:58:59.804988 6232 leader_election.cc:302] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:58:59.805191 7315 raft_consensus.cc:2781] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.805253 7315 raft_consensus.cc:482] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.805290 7315 raft_consensus.cc:3037] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.806082 7315 raft_consensus.cc:504] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.806398 7315 leader_election.cc:288] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.806713 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f943ecfe59f432c861a54b16c62572f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:58:59.806715 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f943ecfe59f432c861a54b16c62572f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:58:59.806825 6429 raft_consensus.cc:3037] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.806823 6561 raft_consensus.cc:3037] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.807756 6561 raft_consensus.cc:2454] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:58:59.807756 6429 raft_consensus.cc:2454] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:58:59.808032 6232 leader_election.cc:302] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:58:59.808200 7315 raft_consensus.cc:2781] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.808295 7315 raft_consensus.cc:686] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:58:59.808406 7315 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.809819 6151 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.844844 7310 raft_consensus.cc:482] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.844957 7310 raft_consensus.cc:504] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.845535 7310 leader_election.cc:288] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.845963 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:58:59.846097 6561 raft_consensus.cc:2454] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:58:59.846369 6625 leader_election.cc:302] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:58:59.846537 7310 raft_consensus.cc:2781] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.846602 7310 raft_consensus.cc:482] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.846644 7310 raft_consensus.cc:3037] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.847486 7310 raft_consensus.cc:504] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.847815 7310 leader_election.cc:288] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.848127 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:58:59.848228 6561 raft_consensus.cc:3037] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.848484 7289 raft_consensus.cc:482] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.848578 7289 raft_consensus.cc:504] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.849087 7289 leader_election.cc:288] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:58:59.849202 6561 raft_consensus.cc:2454] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:58:59.849391 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:58:59.849560 6691 raft_consensus.cc:2379] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1f36d57661b4429a4be8d10b58d8ff7 in current term 1: Already voted for candidate a97073f89fd44702bccb74a9b76f858d in this term.
I20240503 07:58:59.850023 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:58:59.850070 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:58:59.850131 6429 raft_consensus.cc:2454] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:58:59.850291 6625 leader_election.cc:302] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:58:59.850430 7310 raft_consensus.cc:2781] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.850498 7310 raft_consensus.cc:686] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:58:59.850620 7310 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:58:59.852154 6151 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:58:59.854166 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435f73fb9f384b10b353df19037bcac3" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:58:59.854307 6561 raft_consensus.cc:2379] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1f36d57661b4429a4be8d10b58d8ff7 in current term 1: Already voted for candidate a97073f89fd44702bccb74a9b76f858d in this term.
I20240503 07:58:59.854571 6363 leader_election.cc:302] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1f36d57661b4429a4be8d10b58d8ff7; no voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:58:59.854710 7289 raft_consensus.cc:3037] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.855348 7289 raft_consensus.cc:2726] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:58:59.860848 7357 raft_consensus.cc:482] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:58:59.860934 7357 raft_consensus.cc:504] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.861841 7357 leader_election.cc:288] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:58:59.865003 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a63d4f2c744129a47ba3c4b7ef4f1d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:58:59.865118 6429 raft_consensus.cc:2454] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:58:59.865118 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a63d4f2c744129a47ba3c4b7ef4f1d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:58:59.865238 6691 raft_consensus.cc:2454] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:58:59.865432 6497 leader_election.cc:302] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:58:59.865711 7357 raft_consensus.cc:2781] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:58:59.865787 7357 raft_consensus.cc:482] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:58:59.865833 7357 raft_consensus.cc:3037] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.866497 7357 raft_consensus.cc:504] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.866786 7357 leader_election.cc:288] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:58:59.867086 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a63d4f2c744129a47ba3c4b7ef4f1d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:58:59.867197 6691 raft_consensus.cc:3037] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.867199 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a63d4f2c744129a47ba3c4b7ef4f1d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:58:59.867285 6429 raft_consensus.cc:3037] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:58:59.868130 6429 raft_consensus.cc:2454] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:58:59.868130 6691 raft_consensus.cc:2454] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:58:59.868405 6496 leader_election.cc:302] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:58:59.868587 7357 raft_consensus.cc:2781] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:58:59.868671 7357 raft_consensus.cc:686] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:58:59.868947 7357 consensus_queue.cc:237] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:58:59.870780 6151 catalog_manager.cc:5484] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:00.008598 7396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:00.008848 7396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:00.011809 7396 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:00.085937 6691 raft_consensus.cc:1261] T d6a63d4f2c744129a47ba3c4b7ef4f1d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.086717 6430 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.086716 6429 raft_consensus.cc:1261] T d6a63d4f2c744129a47ba3c4b7ef4f1d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.087329 7310 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.087466 6560 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.087780 7349 consensus_queue.cc:1035] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.087817 7315 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.088136 7349 consensus_queue.cc:1035] T d6a63d4f2c744129a47ba3c4b7ef4f1d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.088310 6561 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.089680 7310 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.091686 6429 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.092085 7315 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.873 ms
time per row: 0.0504364 ms
I20240503 07:59:00.149418 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:00.154465 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 0a144fda14a34046a06e558823f02735 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.154786 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a144fda14a34046a06e558823f02735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.155222 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 0a144fda14a34046a06e558823f02735 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.155463 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a144fda14a34046a06e558823f02735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.155602 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.155614 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.155817 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.155838 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.157310 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 0a144fda14a34046a06e558823f02735 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.157560 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a144fda14a34046a06e558823f02735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.157744 6279 tablet_service.cc:1467] Processing CreateTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.157938 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0992eb4c7a471f9646518dd20e7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.159523 7458 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:00.159683 6410 tablet_service.cc:1467] Processing CreateTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.159888 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0992eb4c7a471f9646518dd20e7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.160272 7458 tablet_bootstrap.cc:654] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.160281 7455 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:00.161221 7455 tablet_bootstrap.cc:654] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.162155 6672 tablet_service.cc:1467] Processing CreateTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.162350 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0992eb4c7a471f9646518dd20e7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.162375 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (DEFAULT_TABLE table=rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.162616 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.163998 7457 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:00.164319 7458 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:00.164440 7458 ts_tablet_manager.cc:1367] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:00.164834 7457 tablet_bootstrap.cc:654] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.164968 7458 raft_consensus.cc:348] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.165092 7458 raft_consensus.cc:374] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.165127 7458 raft_consensus.cc:729] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.165252 7458 consensus_queue.cc:260] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.165570 7458 ts_tablet_manager.cc:1398] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.165711 7458 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:00.166071 7455 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:00.166142 7455 ts_tablet_manager.cc:1367] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:00.166450 7458 tablet_bootstrap.cc:654] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.166538 7455 raft_consensus.cc:348] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.166637 7455 raft_consensus.cc:374] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.166666 7455 raft_consensus.cc:729] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.166771 7455 consensus_queue.cc:260] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.166998 7455 ts_tablet_manager.cc:1398] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.167166 7455 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:00.167263 7458 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:00.167320 7458 ts_tablet_manager.cc:1367] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:00.167654 7458 raft_consensus.cc:348] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.167742 7458 raft_consensus.cc:374] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.167768 7458 raft_consensus.cc:729] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.167871 7458 consensus_queue.cc:260] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.167915 7455 tablet_bootstrap.cc:654] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.168084 7458 ts_tablet_manager.cc:1398] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.169060 7456 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:00.169586 7457 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:00.169724 7457 ts_tablet_manager.cc:1367] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:00.169797 7456 tablet_bootstrap.cc:654] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.170236 7457 raft_consensus.cc:348] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.170328 7455 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:00.170342 7457 raft_consensus.cc:374] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.170372 7457 raft_consensus.cc:729] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.170394 7455 ts_tablet_manager.cc:1367] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:00.170527 7457 consensus_queue.cc:260] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.170764 7455 raft_consensus.cc:348] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.170862 7455 raft_consensus.cc:374] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.170897 7455 raft_consensus.cc:729] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.171039 7455 consensus_queue.cc:260] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.171312 7455 ts_tablet_manager.cc:1398] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.171380 7457 ts_tablet_manager.cc:1398] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:00.171514 7455 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:00.172274 7456 tablet_bootstrap.cc:492] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:00.172358 7456 ts_tablet_manager.cc:1367] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.172794 7455 tablet_bootstrap.cc:654] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.173202 7449 raft_consensus.cc:482] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.173285 7449 raft_consensus.cc:504] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.173592 7449 leader_election.cc:288] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:00.173974 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:00.173993 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:00.174089 6561 raft_consensus.cc:2454] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.174328 6625 leader_election.cc:302] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:00.174378 7456 raft_consensus.cc:348] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.174464 7456 raft_consensus.cc:374] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.174471 7449 raft_consensus.cc:2781] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.174494 7456 raft_consensus.cc:729] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.174520 7449 raft_consensus.cc:482] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.174551 7449 raft_consensus.cc:3037] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:00.174567 6625 leader_election.cc:341] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:00.174621 7456 consensus_queue.cc:260] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.174927 7456 ts_tablet_manager.cc:1398] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:00.175062 7456 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:00.175683 7456 tablet_bootstrap.cc:654] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.175694 7449 raft_consensus.cc:504] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.175976 7449 leader_election.cc:288] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:00.176685 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:00.176842 6561 raft_consensus.cc:3037] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.176827 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
W20240503 07:59:00.177114 6625 leader_election.cc:341] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:00.177650 6561 raft_consensus.cc:2454] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.177819 7455 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:00.177899 7455 ts_tablet_manager.cc:1367] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:00.177932 6625 leader_election.cc:302] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters: c167a0a433d34b74affed13cbf96e302
I20240503 07:59:00.178066 7449 raft_consensus.cc:2781] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.178182 7449 raft_consensus.cc:686] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:00.178257 7455 raft_consensus.cc:348] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.178295 7449 consensus_queue.cc:237] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.178356 7455 raft_consensus.cc:374] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.178387 7455 raft_consensus.cc:729] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.178496 7455 consensus_queue.cc:260] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.178733 7455 ts_tablet_manager.cc:1398] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:00.178999 7456 tablet_bootstrap.cc:492] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:00.179070 7456 ts_tablet_manager.cc:1367] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:00.179436 7456 raft_consensus.cc:348] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.179512 7456 raft_consensus.cc:374] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.179538 7456 raft_consensus.cc:729] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.179559 6148 catalog_manager.cc:5484] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:00.179656 7456 consensus_queue.cc:260] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.179872 7456 ts_tablet_manager.cc:1398] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:00.180007 7456 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:00.180748 7456 tablet_bootstrap.cc:654] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.182826 7456 tablet_bootstrap.cc:492] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:00.182898 7456 ts_tablet_manager.cc:1367] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:00.183212 7456 raft_consensus.cc:348] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.183287 7456 raft_consensus.cc:374] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.183318 7456 raft_consensus.cc:729] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.183423 7456 consensus_queue.cc:260] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.183670 7456 ts_tablet_manager.cc:1398] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:00.336732 7449 raft_consensus.cc:482] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.336861 7449 raft_consensus.cc:504] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.337226 7449 leader_election.cc:288] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.337577 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0992eb4c7a471f9646518dd20e7c26" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:00.337594 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0992eb4c7a471f9646518dd20e7c26" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:00.337711 6299 raft_consensus.cc:2454] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.337714 6429 raft_consensus.cc:2454] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.337987 6625 leader_election.cc:302] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:00.338145 7449 raft_consensus.cc:2781] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.338200 7449 raft_consensus.cc:482] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.338243 7449 raft_consensus.cc:3037] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.338960 7449 raft_consensus.cc:504] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.339246 7449 leader_election.cc:288] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.339540 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0992eb4c7a471f9646518dd20e7c26" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:00.339551 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0992eb4c7a471f9646518dd20e7c26" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:00.339637 6429 raft_consensus.cc:3037] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.339641 6299 raft_consensus.cc:3037] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.340341 6429 raft_consensus.cc:2454] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.340353 6299 raft_consensus.cc:2454] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.340607 6628 leader_election.cc:302] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:00.340811 7449 raft_consensus.cc:2781] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.340909 7449 raft_consensus.cc:686] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:00.341042 7449 consensus_queue.cc:237] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.342536 6155 catalog_manager.cc:5484] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:00.510293 7442 raft_consensus.cc:482] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.510422 7442 raft_consensus.cc:504] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.510843 7442 leader_election.cc:288] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:00.511189 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a144fda14a34046a06e558823f02735" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:00.511189 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a144fda14a34046a06e558823f02735" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:00.511322 6691 raft_consensus.cc:2454] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:00.511322 6299 raft_consensus.cc:2454] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:00.511590 6363 leader_election.cc:302] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:00.511766 7442 raft_consensus.cc:2781] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.511826 7442 raft_consensus.cc:482] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.511859 7442 raft_consensus.cc:3037] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.512856 7442 raft_consensus.cc:504] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.513186 7442 leader_election.cc:288] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:00.513480 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a144fda14a34046a06e558823f02735" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:00.513478 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a144fda14a34046a06e558823f02735" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:00.513583 6691 raft_consensus.cc:3037] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.513593 6299 raft_consensus.cc:3037] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.514237 6691 raft_consensus.cc:2454] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:00.514364 6299 raft_consensus.cc:2454] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:00.514492 6365 leader_election.cc:302] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:00.514629 7442 raft_consensus.cc:2781] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.514710 7442 raft_consensus.cc:686] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:00.514978 7442 consensus_queue.cc:237] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.517064 6155 catalog_manager.cc:5484] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:00.546759 7296 raft_consensus.cc:482] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.546880 7296 raft_consensus.cc:504] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.547235 7296 leader_election.cc:288] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:00.547560 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:00.547559 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e10c6d976ed4a8ca0d68b56d7dc753f" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:00.547704 6561 raft_consensus.cc:2379] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c167a0a433d34b74affed13cbf96e302 in current term 1: Already voted for candidate a97073f89fd44702bccb74a9b76f858d in this term.
I20240503 07:59:00.547971 6232 leader_election.cc:302] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c167a0a433d34b74affed13cbf96e302; no voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:00.548118 7296 raft_consensus.cc:3037] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.548867 7296 raft_consensus.cc:2726] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 07:59:00.618274 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:00.618479 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:00.621390 7471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:00.623961 7449 consensus_queue.cc:1035] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.627966 7449 consensus_queue.cc:1035] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 07:59:00.712304 6429 raft_consensus.cc:1261] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.712482 6299 raft_consensus.cc:1261] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.713187 7449 consensus_queue.cc:1035] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.713456 7449 consensus_queue.cc:1035] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.716703 6299 raft_consensus.cc:1261] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.717077 7442 consensus_queue.cc:1035] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:00.717288 6691 raft_consensus.cc:1261] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:00.718500 7442 consensus_queue.cc:1035] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.475 ms
time per row: 0.0562377 ms
I20240503 07:59:00.773351 6152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:00.778442 6279 tablet_service.cc:1467] Processing CreateTablet for tablet afcd161598db41d7962caca61f63ef77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.778519 6410 tablet_service.cc:1467] Processing CreateTablet for tablet afcd161598db41d7962caca61f63ef77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.778653 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3f03610866454595f4b6b87b7cd9cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.778724 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afcd161598db41d7962caca61f63ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.778724 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afcd161598db41d7962caca61f63ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.778926 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3f03610866454595f4b6b87b7cd9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.779100 6540 tablet_service.cc:1467] Processing CreateTablet for tablet afcd161598db41d7962caca61f63ef77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.779363 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afcd161598db41d7962caca61f63ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.781351 7535 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:00.781433 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 8861b52c306e43d8b0615202cf7d8e35 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.781632 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8861b52c306e43d8b0615202cf7d8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.782016 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 8861b52c306e43d8b0615202cf7d8e35 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.782083 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3f03610866454595f4b6b87b7cd9cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.782223 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8861b52c306e43d8b0615202cf7d8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.782281 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3f03610866454595f4b6b87b7cd9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.782572 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3f03610866454595f4b6b87b7cd9cb (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.782768 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3f03610866454595f4b6b87b7cd9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.784047 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 8861b52c306e43d8b0615202cf7d8e35 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:00.784265 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8861b52c306e43d8b0615202cf7d8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:00.784267 7533 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:00.784616 7536 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:00.785104 7533 tablet_bootstrap.cc:654] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.785446 7536 tablet_bootstrap.cc:654] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.785812 7534 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:00.786332 7536 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:00.786412 7536 ts_tablet_manager.cc:1367] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:00.786664 7533 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:00.786698 7535 tablet_bootstrap.cc:654] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.786734 7533 ts_tablet_manager.cc:1367] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:00.786834 7536 raft_consensus.cc:348] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.786944 7536 raft_consensus.cc:374] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.786972 7536 raft_consensus.cc:729] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.787084 7536 consensus_queue.cc:260] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.787134 7533 raft_consensus.cc:348] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.787226 7533 raft_consensus.cc:374] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.787254 7533 raft_consensus.cc:729] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.787340 7536 ts_tablet_manager.cc:1398] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.787385 7533 consensus_queue.cc:260] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.787498 7536 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:00.787710 7533 ts_tablet_manager.cc:1398] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.787981 7534 tablet_bootstrap.cc:654] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.788208 7536 tablet_bootstrap.cc:654] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.788395 7533 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:00.789114 7535 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:00.789175 7533 tablet_bootstrap.cc:654] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.789204 7535 ts_tablet_manager.cc:1367] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:00.789641 7535 raft_consensus.cc:348] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.789738 7535 raft_consensus.cc:374] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.789768 7535 raft_consensus.cc:729] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.789876 7535 consensus_queue.cc:260] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.790104 7534 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:00.790117 7535 ts_tablet_manager.cc:1398] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.790182 7534 ts_tablet_manager.cc:1367] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:00.790263 7535 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:00.790581 7534 raft_consensus.cc:348] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.790681 7534 raft_consensus.cc:374] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.790716 7534 raft_consensus.cc:729] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.790848 7534 consensus_queue.cc:260] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.791057 7535 tablet_bootstrap.cc:654] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.791096 7534 ts_tablet_manager.cc:1398] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.791246 7534 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:00.792028 7534 tablet_bootstrap.cc:654] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.792063 7535 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:00.792125 7535 ts_tablet_manager.cc:1367] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:00.792176 7536 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:00.792253 7536 ts_tablet_manager.cc:1367] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:00.792516 7535 raft_consensus.cc:348] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.792613 7535 raft_consensus.cc:374] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.792649 7535 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.792675 7536 raft_consensus.cc:348] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.792759 7536 raft_consensus.cc:374] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.792810 7536 raft_consensus.cc:729] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.792747 7535 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.793025 7535 ts_tablet_manager.cc:1398] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.793016 7536 consensus_queue.cc:260] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.793520 7533 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:00.793601 7533 ts_tablet_manager.cc:1367] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:00.793927 7534 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:00.793994 7533 raft_consensus.cc:348] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.794050 7534 ts_tablet_manager.cc:1367] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:00.794090 7533 raft_consensus.cc:374] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.794123 7533 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.794360 7533 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.794576 7534 raft_consensus.cc:348] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.794680 7533 ts_tablet_manager.cc:1398] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:00.794680 7534 raft_consensus.cc:374] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.794737 7534 raft_consensus.cc:729] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.794868 7534 consensus_queue.cc:260] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.795102 7534 ts_tablet_manager.cc:1398] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.796205 7536 ts_tablet_manager.cc:1398] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:00.796383 7536 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:00.797322 7536 tablet_bootstrap.cc:654] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:00.798262 7536 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:00.798324 7536 ts_tablet_manager.cc:1367] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:00.798682 7536 raft_consensus.cc:348] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.798774 7536 raft_consensus.cc:374] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:00.798794 7536 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:00.798904 7536 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.799223 7536 ts_tablet_manager.cc:1398] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:00.808130 7349 raft_consensus.cc:482] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.808210 7349 raft_consensus.cc:504] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.808594 7349 leader_election.cc:288] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.808967 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcd161598db41d7962caca61f63ef77" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:00.809078 6429 raft_consensus.cc:2454] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:00.809315 6497 leader_election.cc:302] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:00.809531 7349 raft_consensus.cc:2781] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.809589 7349 raft_consensus.cc:482] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.809633 7349 raft_consensus.cc:3037] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.810230 7349 raft_consensus.cc:504] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.810544 7349 leader_election.cc:288] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.810854 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcd161598db41d7962caca61f63ef77" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:00.810963 6429 raft_consensus.cc:3037] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.811803 6429 raft_consensus.cc:2454] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:00.812062 6497 leader_election.cc:302] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:00.812222 7349 raft_consensus.cc:2781] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.812331 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcd161598db41d7962caca61f63ef77" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:00.812376 7349 raft_consensus.cc:686] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:00.812374 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afcd161598db41d7962caca61f63ef77" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:00.812430 6299 raft_consensus.cc:2454] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:00.812495 7349 consensus_queue.cc:237] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:00.814129 6151 catalog_manager.cc:5484] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:00.857885 7508 raft_consensus.cc:482] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.857986 7508 raft_consensus.cc:504] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.858283 7508 leader_election.cc:288] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:00.858716 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3f03610866454595f4b6b87b7cd9cb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:00.858716 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3f03610866454595f4b6b87b7cd9cb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:00.858861 6298 raft_consensus.cc:2454] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.858860 6561 raft_consensus.cc:2454] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.859127 6625 leader_election.cc:302] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:00.859278 7508 raft_consensus.cc:2781] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.859344 7508 raft_consensus.cc:482] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.859370 7508 raft_consensus.cc:3037] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.860093 7508 raft_consensus.cc:504] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.860388 7508 leader_election.cc:288] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:00.860589 7508 raft_consensus.cc:482] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:00.860675 7508 raft_consensus.cc:504] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.860705 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3f03610866454595f4b6b87b7cd9cb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:00.860810 6561 raft_consensus.cc:3037] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.860901 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3f03610866454595f4b6b87b7cd9cb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:00.861006 6298 raft_consensus.cc:3037] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.861048 7508 leader_election.cc:288] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.861585 6561 raft_consensus.cc:2454] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.861785 6298 raft_consensus.cc:2454] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.861828 6625 leader_election.cc:302] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:00.861898 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:00.861991 7508 raft_consensus.cc:2781] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.862008 6560 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.862066 7508 raft_consensus.cc:686] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:00.862185 7508 consensus_queue.cc:237] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:00.862347 6625 leader_election.cc:302] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:00.862416 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:00.862530 6429 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:00.862706 7509 raft_consensus.cc:2781] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:00.862751 7509 raft_consensus.cc:482] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:00.862782 7509 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.863612 7509 raft_consensus.cc:504] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.863883 7509 leader_election.cc:288] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:00.864145 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:00.864248 6560 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.864441 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:00.864539 6429 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:00.865085 6560 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.865011 6151 catalog_manager.cc:5484] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:00.865366 6625 leader_election.cc:302] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:00.865401 6429 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:00.865561 7509 raft_consensus.cc:2781] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:00.865654 7509 raft_consensus.cc:686] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:00.865767 7509 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:00.867889 6151 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:00.982829 7552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:00.983162 7552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:00.986294 7552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:01.092202 6429 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.092367 6299 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.092443 6560 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.092526 6298 raft_consensus.cc:3037] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.092944 7508 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.093273 7509 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.093600 6298 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.093899 7349 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.095386 6429 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.096736 7508 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.096948 7349 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.097774 6561 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.099378 7508 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.915 ms
time per row: 0.0629576 ms
I20240503 07:59:01.155083 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:01.160966 6410 tablet_service.cc:1467] Processing CreateTablet for tablet e874c9cd82864973a92fb743e45ccf67 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.160986 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 293067841cc6412f853e4da6df8f6b36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.161247 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874c9cd82864973a92fb743e45ccf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.161247 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293067841cc6412f853e4da6df8f6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.161373 6279 tablet_service.cc:1467] Processing CreateTablet for tablet e874c9cd82864973a92fb743e45ccf67 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.161489 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.161587 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874c9cd82864973a92fb743e45ccf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.161702 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.163229 7535 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:01.163430 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.163527 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 293067841cc6412f853e4da6df8f6b36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.163638 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.163700 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293067841cc6412f853e4da6df8f6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.164002 7535 tablet_bootstrap.cc:654] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.164352 7533 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:01.165035 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 293067841cc6412f853e4da6df8f6b36 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.165263 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293067841cc6412f853e4da6df8f6b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.165045 7533 tablet_bootstrap.cc:654] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.165485 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.165663 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.166244 7533 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:01.166344 7533 ts_tablet_manager.cc:1367] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.166759 7533 raft_consensus.cc:348] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.166862 7533 raft_consensus.cc:374] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.166900 7533 raft_consensus.cc:729] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.167017 7533 consensus_queue.cc:260] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.167271 7533 ts_tablet_manager.cc:1398] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.167430 7533 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:01.167450 6541 tablet_service.cc:1467] Processing CreateTablet for tablet e874c9cd82864973a92fb743e45ccf67 (DEFAULT_TABLE table=rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.167659 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e874c9cd82864973a92fb743e45ccf67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.168190 7533 tablet_bootstrap.cc:654] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.169147 7533 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:01.169257 7533 ts_tablet_manager.cc:1367] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:01.169517 7534 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:01.169701 7533 raft_consensus.cc:348] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.169879 7533 raft_consensus.cc:374] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.169942 7533 raft_consensus.cc:729] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.170043 7533 consensus_queue.cc:260] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.170275 7533 ts_tablet_manager.cc:1398] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.170320 7534 tablet_bootstrap.cc:654] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.170850 7536 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:01.171589 7536 tablet_bootstrap.cc:654] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.172437 7536 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:01.172504 7536 ts_tablet_manager.cc:1367] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.172817 7536 raft_consensus.cc:348] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.172907 7536 raft_consensus.cc:374] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.172935 7536 raft_consensus.cc:729] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.173036 7536 consensus_queue.cc:260] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.173251 7536 ts_tablet_manager.cc:1398] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.173292 7535 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:01.173367 7535 ts_tablet_manager.cc:1367] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:01.173393 7536 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:01.173683 7535 raft_consensus.cc:348] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.173780 7535 raft_consensus.cc:374] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.173808 7535 raft_consensus.cc:729] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.173902 7535 consensus_queue.cc:260] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.173997 7536 tablet_bootstrap.cc:654] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.174142 7535 ts_tablet_manager.cc:1398] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.174290 7535 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:01.175010 7535 tablet_bootstrap.cc:654] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.175530 7534 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:01.175606 7534 ts_tablet_manager.cc:1367] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:01.176025 7534 raft_consensus.cc:348] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.176126 7534 raft_consensus.cc:374] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.176162 7534 raft_consensus.cc:729] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.176259 7534 consensus_queue.cc:260] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.176492 7534 ts_tablet_manager.cc:1398] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.176636 7534 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:01.176721 7535 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:01.176826 7535 ts_tablet_manager.cc:1367] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:01.177196 7535 raft_consensus.cc:348] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.177284 7535 raft_consensus.cc:374] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.177312 7535 raft_consensus.cc:729] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.177407 7535 consensus_queue.cc:260] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.177631 7535 ts_tablet_manager.cc:1398] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.178303 7534 tablet_bootstrap.cc:654] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.178344 7536 tablet_bootstrap.cc:492] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:01.178428 7536 ts_tablet_manager.cc:1367] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:01.178802 7536 raft_consensus.cc:348] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.178901 7536 raft_consensus.cc:374] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.178938 7536 raft_consensus.cc:729] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.179047 7536 consensus_queue.cc:260] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.179271 7536 ts_tablet_manager.cc:1398] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.179710 7534 tablet_bootstrap.cc:492] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:01.179792 7534 ts_tablet_manager.cc:1367] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:01.180156 7534 raft_consensus.cc:348] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.180233 7534 raft_consensus.cc:374] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.180261 7534 raft_consensus.cc:729] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.180367 7534 consensus_queue.cc:260] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.180665 7534 ts_tablet_manager.cc:1398] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.180851 7534 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:01.181929 7534 tablet_bootstrap.cc:654] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.182893 7534 tablet_bootstrap.cc:492] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:01.182915 7295 raft_consensus.cc:482] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.182971 7534 ts_tablet_manager.cc:1367] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.182989 7295 raft_consensus.cc:504] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.183306 7295 leader_election.cc:288] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:01.183329 7534 raft_consensus.cc:348] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.183413 7534 raft_consensus.cc:374] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.183442 7534 raft_consensus.cc:729] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.183630 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293067841cc6412f853e4da6df8f6b36" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:01.183738 6430 raft_consensus.cc:2454] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:01.183681 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293067841cc6412f853e4da6df8f6b36" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:01.183980 6628 leader_election.cc:302] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.184007 6299 raft_consensus.cc:2454] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:01.183557 7534 consensus_queue.cc:260] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.184161 7295 raft_consensus.cc:2781] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.184221 7295 raft_consensus.cc:482] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.184253 7295 raft_consensus.cc:3037] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.184322 7534 ts_tablet_manager.cc:1398] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.185093 7295 raft_consensus.cc:504] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.185397 7295 leader_election.cc:288] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:01.185685 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293067841cc6412f853e4da6df8f6b36" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:01.185685 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293067841cc6412f853e4da6df8f6b36" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:01.185778 6430 raft_consensus.cc:3037] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.185778 6299 raft_consensus.cc:3037] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.186573 6430 raft_consensus.cc:2454] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:01.186573 6299 raft_consensus.cc:2454] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:01.186839 6628 leader_election.cc:302] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.186995 7295 raft_consensus.cc:2781] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.187072 7295 raft_consensus.cc:686] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:01.187183 7295 consensus_queue.cc:237] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.188489 6151 catalog_manager.cc:5484] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:01.212242 7521 raft_consensus.cc:482] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.212324 7521 raft_consensus.cc:504] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.212630 7521 leader_election.cc:288] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:01.212967 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874c9cd82864973a92fb743e45ccf67" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:01.213016 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874c9cd82864973a92fb743e45ccf67" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:01.213068 6560 raft_consensus.cc:2454] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.213135 6299 raft_consensus.cc:2454] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.213325 6363 leader_election.cc:302] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.213479 7521 raft_consensus.cc:2781] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.213532 7521 raft_consensus.cc:482] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.213567 7521 raft_consensus.cc:3037] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.214202 7521 raft_consensus.cc:504] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.214468 7521 leader_election.cc:288] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:01.214778 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874c9cd82864973a92fb743e45ccf67" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:01.214881 6560 raft_consensus.cc:3037] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.214877 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e874c9cd82864973a92fb743e45ccf67" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:01.215018 6299 raft_consensus.cc:3037] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.215567 6560 raft_consensus.cc:2454] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.215664 6299 raft_consensus.cc:2454] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.215975 6363 leader_election.cc:302] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.216146 7521 raft_consensus.cc:2781] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.216238 7521 raft_consensus.cc:686] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:01.216388 7521 consensus_queue.cc:237] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:01.217815 6154 catalog_manager.cc:5484] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:01.472738 7526 raft_consensus.cc:482] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.472884 7526 raft_consensus.cc:504] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.473271 7526 leader_election.cc:288] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.473625 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ba39eb4cb245e5bc9c8bea6b1acb09" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:01.473752 6691 raft_consensus.cc:2454] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:01.473759 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ba39eb4cb245e5bc9c8bea6b1acb09" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:01.473910 6560 raft_consensus.cc:2454] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:01.474030 6234 leader_election.cc:302] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:01.474192 7526 raft_consensus.cc:2781] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.474260 7526 raft_consensus.cc:482] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.474303 7526 raft_consensus.cc:3037] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.475030 7526 raft_consensus.cc:504] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.475302 7526 leader_election.cc:288] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.475600 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ba39eb4cb245e5bc9c8bea6b1acb09" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:01.475630 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ba39eb4cb245e5bc9c8bea6b1acb09" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:01.475716 6560 raft_consensus.cc:3037] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.475715 6691 raft_consensus.cc:3037] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.476584 6560 raft_consensus.cc:2454] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:01.476584 6691 raft_consensus.cc:2454] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:01.476850 6232 leader_election.cc:302] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:01.476997 7526 raft_consensus.cc:2781] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.477137 7526 raft_consensus.cc:686] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:01.477269 7526 consensus_queue.cc:237] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:01.478689 6154 catalog_manager.cc:5484] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:01.568246 7295 consensus_queue.cc:1035] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.571621 7295 consensus_queue.cc:1035] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:01.630942 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:01.631136 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:01.634148 7603 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 07:59:01.669014 7521 consensus_queue.cc:1035] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.674713 7289 consensus_queue.cc:1035] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.727758 6560 raft_consensus.cc:1261] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.728127 7526 consensus_queue.cc:1035] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:01.731190 6691 raft_consensus.cc:1261] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:01.731541 7526 consensus_queue.cc:1035] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.7836 ms
time per row: 0.0478918 ms
I20240503 07:59:01.768595 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:01.773777 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 974c815ff8e94190a7bf28444cb3cd9f (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.773872 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 974c815ff8e94190a7bf28444cb3cd9f (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.774077 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974c815ff8e94190a7bf28444cb3cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.774145 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974c815ff8e94190a7bf28444cb3cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.774153 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 5631f3e97fe04013a2a7140f44260003 (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.774343 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5631f3e97fe04013a2a7140f44260003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.774438 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 974c815ff8e94190a7bf28444cb3cd9f (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.774667 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974c815ff8e94190a7bf28444cb3cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.776259 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 5631f3e97fe04013a2a7140f44260003 (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.776341 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 5d95cc5335534f02aa62b00fb8b9c33c (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.776579 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d95cc5335534f02aa62b00fb8b9c33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.776542 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 5d95cc5335534f02aa62b00fb8b9c33c (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.776742 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 5d95cc5335534f02aa62b00fb8b9c33c (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.776953 7661 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:01.777021 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d95cc5335534f02aa62b00fb8b9c33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.777170 7662 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:01.777774 7661 tablet_bootstrap.cc:654] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.778031 7662 tablet_bootstrap.cc:654] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.778453 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 5631f3e97fe04013a2a7140f44260003 (DEFAULT_TABLE table=rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:01.778651 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5631f3e97fe04013a2a7140f44260003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.778872 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d95cc5335534f02aa62b00fb8b9c33c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.779009 7662 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:01.779039 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5631f3e97fe04013a2a7140f44260003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:01.779078 7662 ts_tablet_manager.cc:1367] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.779484 7662 raft_consensus.cc:348] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.779587 7662 raft_consensus.cc:374] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.779616 7662 raft_consensus.cc:729] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.779752 7662 consensus_queue.cc:260] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.780949 7661 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:01.781026 7661 ts_tablet_manager.cc:1367] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:01.781455 7662 ts_tablet_manager.cc:1398] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.781440 7661 raft_consensus.cc:348] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.781528 7661 raft_consensus.cc:374] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.781558 7661 raft_consensus.cc:729] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.781610 7662 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:01.781673 7661 consensus_queue.cc:260] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.781929 7661 ts_tablet_manager.cc:1398] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.782112 7661 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:01.782609 7662 tablet_bootstrap.cc:654] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.782689 7665 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:01.782858 7661 tablet_bootstrap.cc:654] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.783524 7665 tablet_bootstrap.cc:654] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.783646 7662 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:01.783706 7666 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:01.783767 7662 ts_tablet_manager.cc:1367] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:01.784142 7662 raft_consensus.cc:348] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.784241 7662 raft_consensus.cc:374] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.784276 7662 raft_consensus.cc:729] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.784420 7662 consensus_queue.cc:260] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.784569 7666 tablet_bootstrap.cc:654] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.785020 7665 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:01.784746 7662 ts_tablet_manager.cc:1398] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.785241 7662 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:01.785272 7665 ts_tablet_manager.cc:1367] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:01.785584 7661 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:01.785667 7661 ts_tablet_manager.cc:1367] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:01.785874 7665 raft_consensus.cc:348] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.786020 7662 tablet_bootstrap.cc:654] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.786024 7665 raft_consensus.cc:374] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.786058 7661 raft_consensus.cc:348] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.786170 7661 raft_consensus.cc:374] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.786208 7661 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.786142 7665 raft_consensus.cc:729] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.786324 7661 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.786571 7661 ts_tablet_manager.cc:1398] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.786597 7665 consensus_queue.cc:260] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.786895 7665 ts_tablet_manager.cc:1398] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.787210 7666 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:01.787297 7666 ts_tablet_manager.cc:1367] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:01.787475 7665 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:01.787720 7666 raft_consensus.cc:348] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.787827 7666 raft_consensus.cc:374] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.787860 7666 raft_consensus.cc:729] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.787989 7666 consensus_queue.cc:260] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.788242 7666 ts_tablet_manager.cc:1398] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.788429 7662 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:01.788519 7662 ts_tablet_manager.cc:1367] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:01.789062 7665 tablet_bootstrap.cc:654] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.789700 7662 raft_consensus.cc:348] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.789795 7662 raft_consensus.cc:374] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.789826 7662 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.789923 7662 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.790171 7662 ts_tablet_manager.cc:1398] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.792222 7665 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:01.792287 7665 ts_tablet_manager.cc:1367] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:01.792656 7665 raft_consensus.cc:348] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.792737 7665 raft_consensus.cc:374] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.792898 7665 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.793079 7665 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.793296 7665 ts_tablet_manager.cc:1398] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:01.793478 7665 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:01.794186 7665 tablet_bootstrap.cc:654] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:01.795152 7665 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:01.795217 7665 ts_tablet_manager.cc:1367] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:01.795539 7665 raft_consensus.cc:348] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.795637 7665 raft_consensus.cc:374] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:01.795678 7665 raft_consensus.cc:729] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:01.795783 7665 consensus_queue.cc:260] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.796003 7665 ts_tablet_manager.cc:1398] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:01.857241 7548 raft_consensus.cc:482] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.857352 7548 raft_consensus.cc:504] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.857707 7548 leader_election.cc:288] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:01.858038 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d95cc5335534f02aa62b00fb8b9c33c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:01.858069 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d95cc5335534f02aa62b00fb8b9c33c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:01.858182 6430 raft_consensus.cc:2454] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:01.858182 6299 raft_consensus.cc:2454] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:01.858448 6494 leader_election.cc:302] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:01.858592 7548 raft_consensus.cc:2781] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.858649 7548 raft_consensus.cc:482] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.858675 7548 raft_consensus.cc:3037] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.859438 7548 raft_consensus.cc:504] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.859736 7548 leader_election.cc:288] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:01.860013 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d95cc5335534f02aa62b00fb8b9c33c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:01.860035 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d95cc5335534f02aa62b00fb8b9c33c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:01.860103 6299 raft_consensus.cc:3037] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.860152 6430 raft_consensus.cc:3037] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.860932 6299 raft_consensus.cc:2454] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:01.861016 6430 raft_consensus.cc:2454] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:01.861253 6494 leader_election.cc:302] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:01.861392 7548 raft_consensus.cc:2781] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.861555 7548 raft_consensus.cc:686] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:01.861676 7548 consensus_queue.cc:237] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.863166 6154 catalog_manager.cc:5484] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:01.883342 7289 raft_consensus.cc:482] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.883450 7289 raft_consensus.cc:504] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.883827 7289 leader_election.cc:288] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.884159 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:01.884169 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:01.884280 6560 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.884285 6691 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.884560 6363 leader_election.cc:302] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.884716 7289 raft_consensus.cc:2781] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.884793 7289 raft_consensus.cc:482] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.884841 7289 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.885562 7289 raft_consensus.cc:504] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.885840 7289 leader_election.cc:288] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.886150 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:01.886250 6560 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.886458 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:01.886555 6691 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.887351 6691 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.887598 6365 leader_election.cc:302] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.887745 7289 raft_consensus.cc:2781] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.887813 7289 raft_consensus.cc:686] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:01.887941 7289 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.889570 6154 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:01.910979 7289 raft_consensus.cc:482] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:01.911073 7289 raft_consensus.cc:504] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.911375 7289 leader_election.cc:288] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.911728 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974c815ff8e94190a7bf28444cb3cd9f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:01.911790 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974c815ff8e94190a7bf28444cb3cd9f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:01.911887 6691 raft_consensus.cc:2454] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.911900 6561 raft_consensus.cc:2454] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:01.912137 6363 leader_election.cc:302] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.912288 7289 raft_consensus.cc:2781] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:01.912343 7289 raft_consensus.cc:482] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:01.912376 7289 raft_consensus.cc:3037] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.913415 6560 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.941515 7289 raft_consensus.cc:504] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.942076 7289 leader_election.cc:288] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:01.942389 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974c815ff8e94190a7bf28444cb3cd9f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:01.942392 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974c815ff8e94190a7bf28444cb3cd9f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:01.942497 6560 raft_consensus.cc:3037] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.942510 6691 raft_consensus.cc:3037] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:01.943174 6560 raft_consensus.cc:2454] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.943341 6691 raft_consensus.cc:2454] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:01.943436 6363 leader_election.cc:302] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:01.943619 7289 raft_consensus.cc:2781] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:01.943719 7289 raft_consensus.cc:686] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:01.943856 7289 consensus_queue.cc:237] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:01.945513 6154 catalog_manager.cc:5484] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:02.078485 7678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:02.078701 7678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:02.081655 7678 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 07:59:02.172755 6560 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.173494 7289 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.174263 6691 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.174857 7521 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.175837 6299 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.175963 6430 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.176203 7548 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.180816 6692 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.181718 7548 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.181758 6561 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.182292 7289 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.188994 7521 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.227183 7629 mvcc.cc:205] Tried to move back new op lower bound from 7023505990353813504 to 7023505989173891072. Current Snapshot: MvccSnapshot[applied={T|T < 7023505990353813504 or (T in {7023505990353813504})}]
INSERT report
rows total: 2000
time total: 116.404 ms
time per row: 0.0582019 ms
I20240503 07:59:02.238632 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:02.243234 6671 tablet_service.cc:1467] Processing CreateTablet for tablet fd67fb07203e4e99919ec0317bdf077d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.243366 6410 tablet_service.cc:1467] Processing CreateTablet for tablet fd67fb07203e4e99919ec0317bdf077d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.243515 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd67fb07203e4e99919ec0317bdf077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.243613 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd67fb07203e4e99919ec0317bdf077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.243721 6278 tablet_service.cc:1467] Processing CreateTablet for tablet fd67fb07203e4e99919ec0317bdf077d (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.243945 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd67fb07203e4e99919ec0317bdf077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.244012 6277 tablet_service.cc:1467] Processing CreateTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.244215 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f08a2db2da42a8b0d72ec82daa8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.244546 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 5710dff79184444797bb46d981ae6f55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.244710 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710dff79184444797bb46d981ae6f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.244640 6539 tablet_service.cc:1467] Processing CreateTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.244933 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f08a2db2da42a8b0d72ec82daa8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.245656 7662 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:02.246385 7662 tablet_bootstrap.cc:654] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.246626 7665 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:02.247165 6672 tablet_service.cc:1467] Processing CreateTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.247280 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 5710dff79184444797bb46d981ae6f55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.247351 7662 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:02.247406 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f08a2db2da42a8b0d72ec82daa8346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.247418 7662 ts_tablet_manager.cc:1367] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.247447 7665 tablet_bootstrap.cc:654] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.247509 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710dff79184444797bb46d981ae6f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.247771 7662 raft_consensus.cc:348] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.247862 7662 raft_consensus.cc:374] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.247900 7662 raft_consensus.cc:729] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.248016 7662 consensus_queue.cc:260] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.248287 7662 ts_tablet_manager.cc:1398] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.248474 7665 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:02.248554 7665 ts_tablet_manager.cc:1367] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.248957 7665 raft_consensus.cc:348] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.249054 7665 raft_consensus.cc:374] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.249082 7665 raft_consensus.cc:729] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.249083 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 5710dff79184444797bb46d981ae6f55 (DEFAULT_TABLE table=rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.249179 7665 consensus_queue.cc:260] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.249290 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710dff79184444797bb46d981ae6f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.249415 7665 ts_tablet_manager.cc:1398] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.251608 7662 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:02.252393 7662 tablet_bootstrap.cc:654] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.253356 7662 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:02.253427 7662 ts_tablet_manager.cc:1367] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.253773 7662 raft_consensus.cc:348] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.253862 7662 raft_consensus.cc:374] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.253886 7662 raft_consensus.cc:729] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.253980 7662 consensus_queue.cc:260] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.254184 7662 ts_tablet_manager.cc:1398] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.259609 7666 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:02.260404 7666 tablet_bootstrap.cc:654] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.261576 7666 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:02.261656 7666 ts_tablet_manager.cc:1367] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.261971 7666 raft_consensus.cc:348] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.262042 7666 raft_consensus.cc:374] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.262060 7666 raft_consensus.cc:729] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.262161 7666 consensus_queue.cc:260] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.262409 7666 ts_tablet_manager.cc:1398] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.262580 7666 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:02.263216 7666 tablet_bootstrap.cc:654] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.264164 7666 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:02.264245 7666 ts_tablet_manager.cc:1367] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.264626 7666 raft_consensus.cc:348] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.264720 7666 raft_consensus.cc:374] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.264750 7666 raft_consensus.cc:729] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.264884 7666 consensus_queue.cc:260] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.265204 7666 ts_tablet_manager.cc:1398] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.265352 7666 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:02.266266 7666 tablet_bootstrap.cc:654] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.266417 7661 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:02.267156 7661 tablet_bootstrap.cc:654] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.268128 7661 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:02.268208 7661 ts_tablet_manager.cc:1367] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.268565 7661 raft_consensus.cc:348] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.268636 7661 raft_consensus.cc:374] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.268654 7661 raft_consensus.cc:729] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.268745 7661 consensus_queue.cc:260] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.268981 7661 ts_tablet_manager.cc:1398] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.269114 7661 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:02.269402 7666 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:02.269464 7666 ts_tablet_manager.cc:1367] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:02.269758 7661 tablet_bootstrap.cc:654] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.269747 7666 raft_consensus.cc:348] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.269807 7666 raft_consensus.cc:374] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.269826 7666 raft_consensus.cc:729] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.269910 7666 consensus_queue.cc:260] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.270112 7666 ts_tablet_manager.cc:1398] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.271016 7661 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:02.271088 7661 ts_tablet_manager.cc:1367] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.271459 7661 raft_consensus.cc:348] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.271535 7661 raft_consensus.cc:374] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.271562 7661 raft_consensus.cc:729] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.271661 7661 consensus_queue.cc:260] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.271898 7661 ts_tablet_manager.cc:1398] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.272131 7661 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:02.273223 7661 tablet_bootstrap.cc:654] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.274145 7661 tablet_bootstrap.cc:492] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:02.274217 7661 ts_tablet_manager.cc:1367] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:02.274578 7661 raft_consensus.cc:348] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.274655 7661 raft_consensus.cc:374] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.274683 7661 raft_consensus.cc:729] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.275059 7661 consensus_queue.cc:260] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.275390 7661 ts_tablet_manager.cc:1398] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:02.341415 7449 raft_consensus.cc:482] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.341521 7449 raft_consensus.cc:504] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.341825 7449 leader_election.cc:288] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:02.342177 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f08a2db2da42a8b0d72ec82daa8346" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:02.342206 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f08a2db2da42a8b0d72ec82daa8346" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:02.342310 6299 raft_consensus.cc:2454] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.342315 6560 raft_consensus.cc:2454] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.342594 6625 leader_election.cc:302] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:02.342765 7449 raft_consensus.cc:2781] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.342823 7449 raft_consensus.cc:482] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.342849 7449 raft_consensus.cc:3037] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.343609 7449 raft_consensus.cc:504] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.343886 7449 leader_election.cc:288] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:02.344167 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f08a2db2da42a8b0d72ec82daa8346" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:02.344226 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f08a2db2da42a8b0d72ec82daa8346" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:02.344267 6299 raft_consensus.cc:3037] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.344333 6560 raft_consensus.cc:3037] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.345012 6299 raft_consensus.cc:2454] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.345049 6560 raft_consensus.cc:2454] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.345283 6625 leader_election.cc:302] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:02.345526 7449 raft_consensus.cc:2781] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.345687 7449 raft_consensus.cc:686] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:02.345815 7449 consensus_queue.cc:237] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.347303 6148 catalog_manager.cc:5484] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:02.347915 7449 raft_consensus.cc:482] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.347995 7449 raft_consensus.cc:504] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.348317 7449 leader_election.cc:288] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:02.348659 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd67fb07203e4e99919ec0317bdf077d" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:02.348834 6429 raft_consensus.cc:2454] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.349049 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd67fb07203e4e99919ec0317bdf077d" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:02.349182 6299 raft_consensus.cc:2454] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.349372 6628 leader_election.cc:302] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:02.349558 7449 raft_consensus.cc:2781] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.349704 7449 raft_consensus.cc:482] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.349766 7449 raft_consensus.cc:3037] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.350553 7449 raft_consensus.cc:504] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.350862 7449 leader_election.cc:288] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:02.351135 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd67fb07203e4e99919ec0317bdf077d" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:02.351154 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd67fb07203e4e99919ec0317bdf077d" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:02.351239 6429 raft_consensus.cc:3037] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.351241 6299 raft_consensus.cc:3037] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.352052 6299 raft_consensus.cc:2454] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.352111 6429 raft_consensus.cc:2454] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.352319 6625 leader_election.cc:302] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:02.352471 7449 raft_consensus.cc:2781] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.352552 7449 raft_consensus.cc:686] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:02.352664 7449 consensus_queue.cc:237] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.354084 6148 catalog_manager.cc:5484] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:02.496156 7449 raft_consensus.cc:482] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:02.496301 7449 raft_consensus.cc:504] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.496692 7449 leader_election.cc:288] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:02.497046 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710dff79184444797bb46d981ae6f55" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:02.497179 6299 raft_consensus.cc:2454] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.497136 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710dff79184444797bb46d981ae6f55" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:02.497278 6429 raft_consensus.cc:2454] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:02.497459 6625 leader_election.cc:302] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:02.497601 7449 raft_consensus.cc:2781] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:02.497654 7449 raft_consensus.cc:482] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:02.497699 7449 raft_consensus.cc:3037] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.498430 7449 raft_consensus.cc:504] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.498740 7449 leader_election.cc:288] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:02.499078 6299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710dff79184444797bb46d981ae6f55" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:02.499176 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5710dff79184444797bb46d981ae6f55" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:02.499209 6299 raft_consensus.cc:3037] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.499286 6429 raft_consensus.cc:3037] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:02.500057 6299 raft_consensus.cc:2454] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.500150 6429 raft_consensus.cc:2454] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:02.500355 6625 leader_election.cc:302] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:02.500504 7449 raft_consensus.cc:2781] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:02.500586 7449 raft_consensus.cc:686] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:02.500711 7449 consensus_queue.cc:237] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:02.502300 6148 catalog_manager.cc:5484] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:02.647521 7728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:02.647709 7728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:02.650631 7728 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 07:59:02.733510 7449 consensus_queue.cc:1035] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.734181 6429 raft_consensus.cc:1261] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.734546 7449 consensus_queue.cc:1035] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.735071 6299 raft_consensus.cc:1261] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.735117 6298 raft_consensus.cc:1261] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.735611 7449 consensus_queue.cc:1035] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.735883 7449 consensus_queue.cc:1035] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.739006 6297 raft_consensus.cc:1261] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.739691 7724 consensus_queue.cc:1035] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.746798 6430 raft_consensus.cc:1261] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:02.746990 7756 mvcc.cc:205] Tried to move back new op lower bound from 7023505992636203008 to 7023505991684198400. Current Snapshot: MvccSnapshot[applied={T|T < 7023505992636203008}]
I20240503 07:59:02.747417 7449 consensus_queue.cc:1035] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:02.754046 7773 mvcc.cc:205] Tried to move back new op lower bound from 7023505992649314304 to 7023505991077646336. Current Snapshot: MvccSnapshot[applied={T|T < 7023505992649314304}]
I20240503 07:59:02.766757 7765 mvcc.cc:205] Tried to move back new op lower bound from 7023505992649314304 to 7023505991077646336. Current Snapshot: MvccSnapshot[applied={T|T < 7023505992649314304}]
INSERT report
rows total: 2000
time total: 107.836 ms
time per row: 0.0539182 ms
I20240503 07:59:02.799150 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:02.804867 6277 tablet_service.cc:1467] Processing CreateTablet for tablet c7e3555fb9214a40bd8dbf937b87cbaa (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.804948 6410 tablet_service.cc:1467] Processing CreateTablet for tablet c7e3555fb9214a40bd8dbf937b87cbaa (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.805168 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e3555fb9214a40bd8dbf937b87cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.805224 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e3555fb9214a40bd8dbf937b87cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.806078 6672 tablet_service.cc:1467] Processing CreateTablet for tablet a2762b2e6de842bb8c898aea5b3910ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.806142 6670 tablet_service.cc:1467] Processing CreateTablet for tablet edf2c780d0d64db79dca88eac77d4478 (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.806296 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2762b2e6de842bb8c898aea5b3910ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.806456 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf2c780d0d64db79dca88eac77d4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.808514 7785 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:02.808573 6410 tablet_service.cc:1467] Processing CreateTablet for tablet edf2c780d0d64db79dca88eac77d4478 (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.808813 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf2c780d0d64db79dca88eac77d4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.809358 6541 tablet_service.cc:1467] Processing CreateTablet for tablet edf2c780d0d64db79dca88eac77d4478 (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.809492 7785 tablet_bootstrap.cc:654] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.809368 6540 tablet_service.cc:1467] Processing CreateTablet for tablet a2762b2e6de842bb8c898aea5b3910ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.809633 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf2c780d0d64db79dca88eac77d4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.809908 6277 tablet_service.cc:1467] Processing CreateTablet for tablet a2762b2e6de842bb8c898aea5b3910ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.810108 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2762b2e6de842bb8c898aea5b3910ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.810525 6539 tablet_service.cc:1467] Processing CreateTablet for tablet c7e3555fb9214a40bd8dbf937b87cbaa (DEFAULT_TABLE table=rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:02.810736 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e3555fb9214a40bd8dbf937b87cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.811645 7789 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:02.811796 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2762b2e6de842bb8c898aea5b3910ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:02.813666 7786 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:02.813776 7787 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:02.814401 7786 tablet_bootstrap.cc:654] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.814465 7787 tablet_bootstrap.cc:654] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.815259 7786 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:02.815333 7786 ts_tablet_manager.cc:1367] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.815635 7785 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:02.815706 7785 ts_tablet_manager.cc:1367] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:02.815744 7786 raft_consensus.cc:348] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.815840 7786 raft_consensus.cc:374] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.815872 7786 raft_consensus.cc:729] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.815975 7786 consensus_queue.cc:260] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.816062 7785 raft_consensus.cc:348] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.816144 7785 raft_consensus.cc:374] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.816172 7785 raft_consensus.cc:729] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.816234 7786 ts_tablet_manager.cc:1398] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.816303 7785 consensus_queue.cc:260] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.816387 7786 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:02.816546 7785 ts_tablet_manager.cc:1398] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.816681 7785 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:02.817095 7786 tablet_bootstrap.cc:654] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.817528 7785 tablet_bootstrap.cc:654] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.818637 7786 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:02.818708 7786 ts_tablet_manager.cc:1367] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.818873 7787 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:02.818944 7787 ts_tablet_manager.cc:1367] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:02.819070 7786 raft_consensus.cc:348] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.819152 7786 raft_consensus.cc:374] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.819180 7786 raft_consensus.cc:729] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.819275 7786 consensus_queue.cc:260] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.819491 7786 ts_tablet_manager.cc:1398] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.819547 7787 raft_consensus.cc:348] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.819654 7787 raft_consensus.cc:374] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.819691 7787 raft_consensus.cc:729] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.819802 7787 consensus_queue.cc:260] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.822470 7785 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:02.822535 7787 ts_tablet_manager.cc:1398] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:02.822613 7789 tablet_bootstrap.cc:654] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.822634 7785 ts_tablet_manager.cc:1367] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:02.823118 7785 raft_consensus.cc:348] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.823263 7785 raft_consensus.cc:374] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.823282 7787 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:02.823346 7785 raft_consensus.cc:729] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.824162 7787 tablet_bootstrap.cc:654] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.823519 7785 consensus_queue.cc:260] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.825007 7785 ts_tablet_manager.cc:1398] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:02.825767 7787 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:02.825850 7787 ts_tablet_manager.cc:1367] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:02.826221 7787 raft_consensus.cc:348] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.826313 7787 raft_consensus.cc:374] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.826339 7787 raft_consensus.cc:729] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.826444 7787 consensus_queue.cc:260] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.826680 7787 ts_tablet_manager.cc:1398] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.833843 7789 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:02.833933 7789 ts_tablet_manager.cc:1367] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240503 07:59:02.834331 7789 raft_consensus.cc:348] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.834434 7789 raft_consensus.cc:374] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.834465 7789 raft_consensus.cc:729] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.834576 7789 consensus_queue.cc:260] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.834834 7789 ts_tablet_manager.cc:1398] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:02.835018 7789 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:02.835803 7789 tablet_bootstrap.cc:654] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.837177 7789 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:02.837306 7789 ts_tablet_manager.cc:1367] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:02.837774 7789 raft_consensus.cc:348] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.837920 7789 raft_consensus.cc:374] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.837994 7789 raft_consensus.cc:729] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.838150 7789 consensus_queue.cc:260] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:02.838447 7789 ts_tablet_manager.cc:1398] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:02.838616 7789 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:02.839403 7789 tablet_bootstrap.cc:654] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:02.840390 7789 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:02.840478 7789 ts_tablet_manager.cc:1367] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:02.840925 7789 raft_consensus.cc:348] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.841017 7789 raft_consensus.cc:374] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:02.841044 7789 raft_consensus.cc:729] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:02.841145 7789 consensus_queue.cc:260] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:02.841382 7789 ts_tablet_manager.cc:1398] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.006691 7703 raft_consensus.cc:482] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.006809 7703 raft_consensus.cc:504] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.007174 7703 leader_election.cc:288] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:03.007529 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:03.007609 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:03.007673 6297 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.007737 6560 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.008028 6363 leader_election.cc:302] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.008200 7703 raft_consensus.cc:2781] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.008270 7703 raft_consensus.cc:482] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.008303 7703 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.011263 7703 raft_consensus.cc:504] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.011582 7703 leader_election.cc:288] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:03.011891 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:03.011960 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:03.012012 6297 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.012054 6560 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.012754 6297 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.012909 6560 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.013082 6363 leader_election.cc:302] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.013240 7703 raft_consensus.cc:2781] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.013495 7703 raft_consensus.cc:686] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:03.013626 7703 consensus_queue.cc:237] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.015156 6150 catalog_manager.cc:5484] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:03.032094 7703 raft_consensus.cc:482] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.032187 7703 raft_consensus.cc:504] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.032624 7703 leader_election.cc:288] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:03.032817 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf2c780d0d64db79dca88eac77d4478" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:03.032930 6690 raft_consensus.cc:2454] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.033102 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf2c780d0d64db79dca88eac77d4478" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:03.033210 6560 raft_consensus.cc:2454] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.033202 6365 leader_election.cc:302] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.033352 7703 raft_consensus.cc:2781] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.033427 7703 raft_consensus.cc:482] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.033463 7703 raft_consensus.cc:3037] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.034126 7703 raft_consensus.cc:504] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.034418 7703 leader_election.cc:288] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:03.034731 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf2c780d0d64db79dca88eac77d4478" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:03.034845 6690 raft_consensus.cc:3037] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.034894 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf2c780d0d64db79dca88eac77d4478" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:03.035004 6560 raft_consensus.cc:3037] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.035601 6690 raft_consensus.cc:2454] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.035815 6560 raft_consensus.cc:2454] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.035838 6365 leader_election.cc:302] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.035979 7703 raft_consensus.cc:2781] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.036051 7703 raft_consensus.cc:686] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:03.036173 7703 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.037703 6150 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:03.070201 7780 raft_consensus.cc:482] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.070288 7780 raft_consensus.cc:504] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.070705 7780 leader_election.cc:288] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.071056 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2762b2e6de842bb8c898aea5b3910ba" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:03.071111 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2762b2e6de842bb8c898aea5b3910ba" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:03.071178 6560 raft_consensus.cc:2454] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:03.071215 6297 raft_consensus.cc:2454] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:03.071455 6625 leader_election.cc:302] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:03.071612 7780 raft_consensus.cc:2781] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.071684 7780 raft_consensus.cc:482] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.071724 7780 raft_consensus.cc:3037] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.072506 7780 raft_consensus.cc:504] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.072856 7780 leader_election.cc:288] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.073072 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2762b2e6de842bb8c898aea5b3910ba" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:03.073132 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2762b2e6de842bb8c898aea5b3910ba" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:03.073181 6560 raft_consensus.cc:3037] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.073233 6297 raft_consensus.cc:3037] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.074061 6297 raft_consensus.cc:2454] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:03.074079 6560 raft_consensus.cc:2454] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:03.074386 6625 leader_election.cc:302] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:03.074563 7780 raft_consensus.cc:2781] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.074676 7780 raft_consensus.cc:686] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:03.074808 7780 consensus_queue.cc:237] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.076478 6150 catalog_manager.cc:5484] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:03.203403 7802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:03.203589 7802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:03.206637 7802 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 07:59:03.274843 6297 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.274859 6561 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.275215 7780 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.275439 7769 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.282006 6560 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.282361 7703 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.283106 6690 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.283591 7703 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.285287 6559 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.285719 6297 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.286126 7798 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.286399 7798 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.18 ms
time per row: 0.05309 ms
I20240503 07:59:03.350404 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:03.355329 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.355609 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cf7d1a310e44b0aee029422b88f0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.355568 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.355824 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cf7d1a310e44b0aee029422b88f0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.356019 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 09c32a1e047844a3b09f2c790d735c9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.356240 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c32a1e047844a3b09f2c790d735c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.357273 7861 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:03.357439 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.357646 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29cf7d1a310e44b0aee029422b88f0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.358081 7861 tablet_bootstrap.cc:654] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.358393 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 62e54fbe89f843168b4ecc21307a6739 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.358564 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e54fbe89f843168b4ecc21307a6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.359302 7861 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:03.359423 7861 ts_tablet_manager.cc:1367] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:03.359853 7861 raft_consensus.cc:348] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.359989 7861 raft_consensus.cc:374] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.360040 7861 raft_consensus.cc:729] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.359988 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 62e54fbe89f843168b4ecc21307a6739 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.359992 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 09c32a1e047844a3b09f2c790d735c9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.360265 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e54fbe89f843168b4ecc21307a6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.360188 7861 consensus_queue.cc:260] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.360337 7862 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:03.360522 7861 ts_tablet_manager.cc:1398] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.360726 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 62e54fbe89f843168b4ecc21307a6739 (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.360986 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62e54fbe89f843168b4ecc21307a6739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.361132 7862 tablet_bootstrap.cc:654] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.362071 7862 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:03.362139 7862 ts_tablet_manager.cc:1367] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.362530 7862 raft_consensus.cc:348] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.362756 7862 raft_consensus.cc:374] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.362783 7861 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:03.362800 7862 raft_consensus.cc:729] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.362910 7862 consensus_queue.cc:260] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.363162 7862 ts_tablet_manager.cc:1398] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.363284 7866 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:03.363304 7862 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:03.363560 7861 tablet_bootstrap.cc:654] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.364105 7866 tablet_bootstrap.cc:654] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.364130 7862 tablet_bootstrap.cc:654] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.364436 7861 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:03.364512 7861 ts_tablet_manager.cc:1367] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:03.364581 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c32a1e047844a3b09f2c790d735c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.364923 7861 raft_consensus.cc:348] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.365020 7861 raft_consensus.cc:374] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.365058 7861 raft_consensus.cc:729] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.365180 7861 consensus_queue.cc:260] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.365417 7861 ts_tablet_manager.cc:1398] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.366919 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 09c32a1e047844a3b09f2c790d735c9b (DEFAULT_TABLE table=rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.367115 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c32a1e047844a3b09f2c790d735c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.368348 7861 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:03.368470 7863 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:03.369213 7861 tablet_bootstrap.cc:654] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.369244 7863 tablet_bootstrap.cc:654] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.370071 7861 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:03.370090 7863 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:03.370134 7861 ts_tablet_manager.cc:1367] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.370154 7863 ts_tablet_manager.cc:1367] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.370442 7861 raft_consensus.cc:348] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.370515 7861 raft_consensus.cc:374] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.370486 7863 raft_consensus.cc:348] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.370544 7861 raft_consensus.cc:729] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.370556 7863 raft_consensus.cc:374] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.370582 7863 raft_consensus.cc:729] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.370659 7861 consensus_queue.cc:260] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.370711 7863 consensus_queue.cc:260] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.370901 7861 ts_tablet_manager.cc:1398] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.370955 7863 ts_tablet_manager.cc:1398] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.371099 7863 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:03.372246 7863 tablet_bootstrap.cc:654] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.373193 7866 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:03.373283 7866 ts_tablet_manager.cc:1367] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:03.373348 7863 tablet_bootstrap.cc:492] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:03.373443 7863 ts_tablet_manager.cc:1367] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:03.373473 7862 tablet_bootstrap.cc:492] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:03.373611 7862 ts_tablet_manager.cc:1367] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:03.373701 7866 raft_consensus.cc:348] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.373796 7866 raft_consensus.cc:374] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.373831 7866 raft_consensus.cc:729] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.373823 7863 raft_consensus.cc:348] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.373932 7863 raft_consensus.cc:374] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.373970 7863 raft_consensus.cc:729] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.373972 7866 consensus_queue.cc:260] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.374233 7866 ts_tablet_manager.cc:1398] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.374449 7866 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:03.374426 7862 raft_consensus.cc:348] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.374599 7862 raft_consensus.cc:374] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.374667 7862 raft_consensus.cc:729] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.374837 7862 consensus_queue.cc:260] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.375206 7866 tablet_bootstrap.cc:654] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.375387 7862 ts_tablet_manager.cc:1398] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.374078 7863 consensus_queue.cc:260] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.376099 7866 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:03.376169 7866 ts_tablet_manager.cc:1367] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.376284 7863 ts_tablet_manager.cc:1398] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:03.376549 7866 raft_consensus.cc:348] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.376638 7866 raft_consensus.cc:374] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.376665 7866 raft_consensus.cc:729] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.376817 7866 consensus_queue.cc:260] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.377054 7866 ts_tablet_manager.cc:1398] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.398566 7798 raft_consensus.cc:482] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.398653 7798 raft_consensus.cc:504] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.398912 7798 leader_election.cc:288] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.399209 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c32a1e047844a3b09f2c790d735c9b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:03.399267 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c32a1e047844a3b09f2c790d735c9b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:03.399318 6560 raft_consensus.cc:2454] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.399381 6297 raft_consensus.cc:2454] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.399575 6363 leader_election.cc:302] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.399732 7798 raft_consensus.cc:2781] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.399789 7798 raft_consensus.cc:482] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.399816 7798 raft_consensus.cc:3037] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.400497 7798 raft_consensus.cc:504] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.400792 7798 leader_election.cc:288] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.401072 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c32a1e047844a3b09f2c790d735c9b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:03.401161 6560 raft_consensus.cc:3037] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.401145 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c32a1e047844a3b09f2c790d735c9b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:03.401252 6297 raft_consensus.cc:3037] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.401824 6560 raft_consensus.cc:2454] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.401909 6297 raft_consensus.cc:2454] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.402102 6363 leader_election.cc:302] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.402247 7798 raft_consensus.cc:2781] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.402328 7798 raft_consensus.cc:686] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:03.402452 7798 consensus_queue.cc:237] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:03.403937 6151 catalog_manager.cc:5484] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:03.511008 7674 raft_consensus.cc:482] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.511127 7674 raft_consensus.cc:504] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.511488 7674 leader_election.cc:288] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.511873 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e54fbe89f843168b4ecc21307a6739" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:03.511870 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e54fbe89f843168b4ecc21307a6739" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:03.512032 6690 raft_consensus.cc:2454] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:03.512029 6297 raft_consensus.cc:2454] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:03.512303 6494 leader_election.cc:302] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:03.512531 7674 raft_consensus.cc:2781] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.512589 7674 raft_consensus.cc:482] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.512624 7674 raft_consensus.cc:3037] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.513447 7674 raft_consensus.cc:504] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.513738 7674 leader_election.cc:288] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:03.514071 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e54fbe89f843168b4ecc21307a6739" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:03.514071 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62e54fbe89f843168b4ecc21307a6739" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:03.514192 6690 raft_consensus.cc:3037] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.514192 6297 raft_consensus.cc:3037] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.515115 6297 raft_consensus.cc:2454] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:03.515115 6690 raft_consensus.cc:2454] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:03.515399 6494 leader_election.cc:302] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:03.515623 7674 raft_consensus.cc:2781] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.515786 7674 raft_consensus.cc:686] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:03.515941 7674 consensus_queue.cc:237] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.517683 6151 catalog_manager.cc:5484] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:03.557444 7798 raft_consensus.cc:482] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.557559 7798 raft_consensus.cc:504] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.557842 7798 leader_election.cc:288] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.558123 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cf7d1a310e44b0aee029422b88f0e7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:03.558151 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cf7d1a310e44b0aee029422b88f0e7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:03.558233 6297 raft_consensus.cc:2454] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.558260 6690 raft_consensus.cc:2454] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:03.558542 6365 leader_election.cc:302] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.558734 7798 raft_consensus.cc:2781] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.558781 7798 raft_consensus.cc:482] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.558805 7798 raft_consensus.cc:3037] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.559602 7798 raft_consensus.cc:504] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.559917 7798 leader_election.cc:288] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.560201 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cf7d1a310e44b0aee029422b88f0e7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:03.560200 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29cf7d1a310e44b0aee029422b88f0e7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:03.560292 6690 raft_consensus.cc:3037] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.560297 6297 raft_consensus.cc:3037] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.561040 6690 raft_consensus.cc:2454] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.561141 6297 raft_consensus.cc:2454] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:03.561296 6365 leader_election.cc:302] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.561473 7798 raft_consensus.cc:2781] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.561570 7798 raft_consensus.cc:686] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:03.561702 7798 consensus_queue.cc:237] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:03.563300 6151 catalog_manager.cc:5484] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:03.701004 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:03.701181 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:03.704020 7879 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 07:59:03.775295 6690 raft_consensus.cc:1261] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.775691 6297 raft_consensus.cc:1261] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.775898 7703 consensus_queue.cc:1035] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.776834 6560 raft_consensus.cc:1261] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.776844 7798 consensus_queue.cc:1035] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.779351 6298 raft_consensus.cc:1261] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.779543 6297 raft_consensus.cc:1261] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.779958 7703 consensus_queue.cc:1035] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.779950 7875 consensus_queue.cc:1035] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.780201 7703 consensus_queue.cc:1035] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:03.782088 6691 raft_consensus.cc:1261] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:03.784654 7674 consensus_queue.cc:1035] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.513 ms
time per row: 0.0552563 ms
I20240503 07:59:03.853720 6152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:03.859220 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 5538a709d8d440cb8f006a163f259389 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.859467 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 82de209205b342cfaac10506fdac9303 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.859529 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5538a709d8d440cb8f006a163f259389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.859737 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82de209205b342cfaac10506fdac9303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.859771 6409 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4259777e5492c904e6d65a36d92ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.860021 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4259777e5492c904e6d65a36d92ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.861199 7863 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:03.861984 7863 tablet_bootstrap.cc:654] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.862162 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 82de209205b342cfaac10506fdac9303 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.862357 6671 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4259777e5492c904e6d65a36d92ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.862433 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82de209205b342cfaac10506fdac9303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.862553 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4259777e5492c904e6d65a36d92ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.862162 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 5538a709d8d440cb8f006a163f259389 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.863198 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5538a709d8d440cb8f006a163f259389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.863255 7863 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:03.863319 7863 ts_tablet_manager.cc:1367] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:03.863688 7863 raft_consensus.cc:348] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.863773 7863 raft_consensus.cc:374] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.863801 7863 raft_consensus.cc:729] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.863911 7863 consensus_queue.cc:260] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.864157 7863 ts_tablet_manager.cc:1398] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.864308 7863 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:03.864928 7862 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:03.865208 7863 tablet_bootstrap.cc:654] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.865746 7862 tablet_bootstrap.cc:654] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.865936 6277 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4259777e5492c904e6d65a36d92ed (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.866180 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4259777e5492c904e6d65a36d92ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.866511 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 82de209205b342cfaac10506fdac9303 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.866523 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 5538a709d8d440cb8f006a163f259389 (DEFAULT_TABLE table=rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:03.866779 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82de209205b342cfaac10506fdac9303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.867131 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5538a709d8d440cb8f006a163f259389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:03.868430 7861 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:03.868697 7866 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:03.869238 7861 tablet_bootstrap.cc:654] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.869256 7862 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:03.869333 7862 ts_tablet_manager.cc:1367] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:03.869550 7866 tablet_bootstrap.cc:654] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.869764 7862 raft_consensus.cc:348] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.869897 7862 raft_consensus.cc:374] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.869933 7862 raft_consensus.cc:729] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.870052 7862 consensus_queue.cc:260] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.870311 7862 ts_tablet_manager.cc:1398] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.870468 7862 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:03.871101 7863 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:03.871176 7863 ts_tablet_manager.cc:1367] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:03.871320 7862 tablet_bootstrap.cc:654] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.871521 7863 raft_consensus.cc:348] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.871596 7863 raft_consensus.cc:374] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.871624 7863 raft_consensus.cc:729] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.871816 7863 consensus_queue.cc:260] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.872440 7861 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:03.872527 7861 ts_tablet_manager.cc:1367] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:03.872529 7863 ts_tablet_manager.cc:1398] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:03.872675 7863 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:03.872988 7861 raft_consensus.cc:348] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.873102 7861 raft_consensus.cc:374] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.873139 7861 raft_consensus.cc:729] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.873256 7861 consensus_queue.cc:260] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.873536 7861 ts_tablet_manager.cc:1398] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.873579 7863 tablet_bootstrap.cc:654] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.874130 7862 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:03.874205 7862 ts_tablet_manager.cc:1367] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:03.874585 7862 raft_consensus.cc:348] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.874672 7862 raft_consensus.cc:374] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.874701 7862 raft_consensus.cc:729] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.874796 7862 consensus_queue.cc:260] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.875034 7862 ts_tablet_manager.cc:1398] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.875118 7866 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:03.875173 7862 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:03.875183 7866 ts_tablet_manager.cc:1367] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:03.875336 7863 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:03.875423 7863 ts_tablet_manager.cc:1367] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:03.875592 7866 raft_consensus.cc:348] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.875686 7866 raft_consensus.cc:374] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.875716 7866 raft_consensus.cc:729] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.875829 7866 consensus_queue.cc:260] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.875840 7863 raft_consensus.cc:348] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.875946 7863 raft_consensus.cc:374] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.875994 7863 raft_consensus.cc:729] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.876096 7866 ts_tablet_manager.cc:1398] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:03.876116 7863 consensus_queue.cc:260] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.876227 7866 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:03.876441 7863 ts_tablet_manager.cc:1398] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:03.877058 7866 tablet_bootstrap.cc:654] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.878350 7866 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:03.878430 7866 ts_tablet_manager.cc:1367] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:03.878829 7866 raft_consensus.cc:348] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.878937 7866 raft_consensus.cc:374] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.878969 7866 raft_consensus.cc:729] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.879058 7866 consensus_queue.cc:260] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.879295 7866 ts_tablet_manager.cc:1398] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.879776 7862 tablet_bootstrap.cc:654] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:03.880932 7862 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:03.881019 7862 ts_tablet_manager.cc:1367] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:03.881350 7862 raft_consensus.cc:348] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.881448 7862 raft_consensus.cc:374] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:03.881481 7862 raft_consensus.cc:729] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:03.881579 7862 consensus_queue.cc:260] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.881839 7862 ts_tablet_manager.cc:1398] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:03.882119 7674 raft_consensus.cc:482] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.882200 7674 raft_consensus.cc:504] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.882547 7674 leader_election.cc:288] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.882784 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5538a709d8d440cb8f006a163f259389" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:03.882824 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5538a709d8d440cb8f006a163f259389" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:03.882882 6430 raft_consensus.cc:2454] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:03.882951 6690 raft_consensus.cc:2454] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:03.883128 6497 leader_election.cc:302] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.883316 7674 raft_consensus.cc:2781] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.883358 7674 raft_consensus.cc:482] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.883391 7674 raft_consensus.cc:3037] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.884039 7674 raft_consensus.cc:504] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.884315 7674 leader_election.cc:288] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.884624 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5538a709d8d440cb8f006a163f259389" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:03.884649 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5538a709d8d440cb8f006a163f259389" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:03.884730 6430 raft_consensus.cc:3037] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.884730 6690 raft_consensus.cc:3037] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.885634 6430 raft_consensus.cc:2454] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:03.885809 6690 raft_consensus.cc:2454] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:03.885907 6497 leader_election.cc:302] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.886122 7674 raft_consensus.cc:2781] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.886204 7674 raft_consensus.cc:686] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:03.886318 7674 consensus_queue.cc:237] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:03.887833 6157 catalog_manager.cc:5484] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:03.944103 7877 raft_consensus.cc:482] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:03.944219 7877 raft_consensus.cc:504] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.944679 7877 leader_election.cc:288] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.944912 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4259777e5492c904e6d65a36d92ed" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:03.945046 6430 raft_consensus.cc:2454] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:03.945035 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4259777e5492c904e6d65a36d92ed" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:03.945158 6690 raft_consensus.cc:2454] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:03.945336 6235 leader_election.cc:302] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.945575 7877 raft_consensus.cc:2781] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:03.945632 7877 raft_consensus.cc:482] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:03.945679 7877 raft_consensus.cc:3037] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.946604 7877 raft_consensus.cc:504] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.946921 7877 leader_election.cc:288] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:03.947214 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4259777e5492c904e6d65a36d92ed" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:03.947222 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4259777e5492c904e6d65a36d92ed" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:03.947328 6430 raft_consensus.cc:3037] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.947372 6690 raft_consensus.cc:3037] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:03.947980 6430 raft_consensus.cc:2454] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:03.948103 6690 raft_consensus.cc:2454] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:03.948273 6235 leader_election.cc:302] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:03.948477 7877 raft_consensus.cc:2781] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:03.948581 7877 raft_consensus.cc:686] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:03.948701 7877 consensus_queue.cc:237] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:03.950332 6157 catalog_manager.cc:5484] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:04.046316 7780 raft_consensus.cc:482] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.046442 7780 raft_consensus.cc:504] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.046808 7780 leader_election.cc:288] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.047155 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82de209205b342cfaac10506fdac9303" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:04.047245 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82de209205b342cfaac10506fdac9303" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:04.047307 6430 raft_consensus.cc:2454] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:04.047417 6561 raft_consensus.cc:2454] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:04.047636 6628 leader_election.cc:302] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.047792 7780 raft_consensus.cc:2781] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.047879 7780 raft_consensus.cc:482] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.047922 7780 raft_consensus.cc:3037] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.048899 7780 raft_consensus.cc:504] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.049233 7780 leader_election.cc:288] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.049539 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82de209205b342cfaac10506fdac9303" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:04.049561 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82de209205b342cfaac10506fdac9303" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:04.049644 6561 raft_consensus.cc:3037] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.049674 6430 raft_consensus.cc:3037] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.050360 6561 raft_consensus.cc:2454] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:04.050376 6430 raft_consensus.cc:2454] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:04.050638 6628 leader_election.cc:302] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.050797 7780 raft_consensus.cc:2781] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.050889 7780 raft_consensus.cc:686] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:04.051057 7780 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.052912 6157 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:04.199218 7928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.199424 7928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.202330 7928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 07:59:04.273775 6430 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.273775 6561 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.274148 7926 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.274622 7926 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.276938 6429 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.278419 7674 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.278991 6690 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.279330 7877 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.279973 6428 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.280310 6690 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.280881 7877 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.289355 7875 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.708 ms
time per row: 0.0553539 ms
I20240503 07:59:04.350899 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:04.356057 6409 tablet_service.cc:1467] Processing CreateTablet for tablet dc995af1acb9459f9096867bc3b075f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.356210 6540 tablet_service.cc:1467] Processing CreateTablet for tablet dc995af1acb9459f9096867bc3b075f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.356321 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc995af1acb9459f9096867bc3b075f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.356593 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc995af1acb9459f9096867bc3b075f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.358444 7863 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:04.358450 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05300c9002408084f0b47d5e47591d (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.358554 7866 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:04.358672 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f05300c9002408084f0b47d5e47591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.358772 6277 tablet_service.cc:1467] Processing CreateTablet for tablet dc995af1acb9459f9096867bc3b075f4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.359033 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc995af1acb9459f9096867bc3b075f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.359302 7866 tablet_bootstrap.cc:654] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.359349 7863 tablet_bootstrap.cc:654] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.360595 7862 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:04.360862 7863 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:04.360949 7863 ts_tablet_manager.cc:1367] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:04.361254 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05300c9002408084f0b47d5e47591d (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.361348 7862 tablet_bootstrap.cc:654] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.361375 7863 raft_consensus.cc:348] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.361485 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f05300c9002408084f0b47d5e47591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.361488 7863 raft_consensus.cc:374] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.361521 7863 raft_consensus.cc:729] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.361642 7863 consensus_queue.cc:260] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.361951 7863 ts_tablet_manager.cc:1398] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.362166 7861 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:04.362787 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.362953 7861 tablet_bootstrap.cc:654] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.363029 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610160f6ba94483a9bdcf966c413e5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.363435 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.363662 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610160f6ba94483a9bdcf966c413e5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.363821 7862 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:04.363896 7862 ts_tablet_manager.cc:1367] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:04.364265 7862 raft_consensus.cc:348] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.364352 7862 raft_consensus.cc:374] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.364382 7862 raft_consensus.cc:729] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.364495 7862 consensus_queue.cc:260] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.364737 7862 ts_tablet_manager.cc:1398] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.365782 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 5f05300c9002408084f0b47d5e47591d (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.365975 7866 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:04.366029 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f05300c9002408084f0b47d5e47591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.366048 7866 ts_tablet_manager.cc:1367] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:59:04.366456 7866 raft_consensus.cc:348] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.366547 7866 raft_consensus.cc:374] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.366557 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.366789 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610160f6ba94483a9bdcf966c413e5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.366569 7866 raft_consensus.cc:729] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.367131 7866 consensus_queue.cc:260] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.367367 7866 ts_tablet_manager.cc:1398] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.367728 7863 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:04.367771 7866 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:04.368430 7863 tablet_bootstrap.cc:654] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.368954 7866 tablet_bootstrap.cc:654] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.370249 7866 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:04.370345 7866 ts_tablet_manager.cc:1367] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:04.370383 7861 tablet_bootstrap.cc:492] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:04.370455 7861 ts_tablet_manager.cc:1367] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:04.370633 7863 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:04.370707 7863 ts_tablet_manager.cc:1367] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:04.370774 7866 raft_consensus.cc:348] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.370891 7866 raft_consensus.cc:374] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.370935 7866 raft_consensus.cc:729] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.370895 7861 raft_consensus.cc:348] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.370985 7861 raft_consensus.cc:374] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.371022 7861 raft_consensus.cc:729] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.371066 7866 consensus_queue.cc:260] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.371110 7863 raft_consensus.cc:348] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.371131 7861 consensus_queue.cc:260] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.371182 7863 raft_consensus.cc:374] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.371208 7863 raft_consensus.cc:729] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.371308 7863 consensus_queue.cc:260] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.371351 7866 ts_tablet_manager.cc:1398] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:04.371398 7861 ts_tablet_manager.cc:1398] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.371560 7861 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:04.371560 7866 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:04.371567 7863 ts_tablet_manager.cc:1398] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.372531 7861 tablet_bootstrap.cc:654] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.373634 7861 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:04.373703 7866 tablet_bootstrap.cc:654] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.373719 7861 ts_tablet_manager.cc:1367] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:04.374125 7861 raft_consensus.cc:348] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.374246 7861 raft_consensus.cc:374] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.374281 7861 raft_consensus.cc:729] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.374428 7861 consensus_queue.cc:260] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.374750 7861 ts_tablet_manager.cc:1398] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.374944 7861 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:04.375686 7861 tablet_bootstrap.cc:654] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.376080 7866 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:04.376152 7866 ts_tablet_manager.cc:1367] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:04.376516 7866 raft_consensus.cc:348] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.376608 7866 raft_consensus.cc:374] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.376624 7861 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:04.376638 7866 raft_consensus.cc:729] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.376696 7861 ts_tablet_manager.cc:1367] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.377058 7866 consensus_queue.cc:260] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.377169 7861 raft_consensus.cc:348] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.377269 7861 raft_consensus.cc:374] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.377303 7861 raft_consensus.cc:729] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.377411 7866 ts_tablet_manager.cc:1398] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:04.377408 7861 consensus_queue.cc:260] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.377679 7861 ts_tablet_manager.cc:1398] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.456619 7914 raft_consensus.cc:482] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.456724 7914 raft_consensus.cc:504] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.457075 7914 leader_election.cc:288] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.457378 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610160f6ba94483a9bdcf966c413e5d1" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:04.457422 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610160f6ba94483a9bdcf966c413e5d1" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:04.457511 6298 raft_consensus.cc:2454] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:04.457551 6561 raft_consensus.cc:2454] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:04.457782 6363 leader_election.cc:302] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.457945 7914 raft_consensus.cc:2781] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.457995 7914 raft_consensus.cc:482] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.458036 7914 raft_consensus.cc:3037] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.458859 7914 raft_consensus.cc:504] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.459146 7914 leader_election.cc:288] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.459439 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610160f6ba94483a9bdcf966c413e5d1" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:04.459507 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610160f6ba94483a9bdcf966c413e5d1" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:04.459549 6298 raft_consensus.cc:3037] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.459612 6561 raft_consensus.cc:3037] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.460413 6298 raft_consensus.cc:2454] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:04.460486 6561 raft_consensus.cc:2454] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:04.460644 6363 leader_election.cc:302] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.460791 7914 raft_consensus.cc:2781] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.460880 7914 raft_consensus.cc:686] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:04.461006 7914 consensus_queue.cc:237] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.463699 6152 catalog_manager.cc:5484] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:04.469501 7875 raft_consensus.cc:482] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.469574 7875 raft_consensus.cc:504] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.469897 7875 leader_election.cc:288] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:04.470201 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f05300c9002408084f0b47d5e47591d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:04.470203 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f05300c9002408084f0b47d5e47591d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:04.470316 6690 raft_consensus.cc:2454] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:04.470307 6298 raft_consensus.cc:2454] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:04.470551 6494 leader_election.cc:302] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:04.470763 7875 raft_consensus.cc:2781] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.470806 7875 raft_consensus.cc:482] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.470840 7875 raft_consensus.cc:3037] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.471535 7875 raft_consensus.cc:504] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.471800 7875 leader_election.cc:288] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:04.472060 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f05300c9002408084f0b47d5e47591d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:04.472091 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f05300c9002408084f0b47d5e47591d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:04.472152 6298 raft_consensus.cc:3037] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.472183 6690 raft_consensus.cc:3037] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.472808 6298 raft_consensus.cc:2454] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:04.472921 6690 raft_consensus.cc:2454] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:04.473048 6494 leader_election.cc:302] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:04.473192 7875 raft_consensus.cc:2781] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.473263 7875 raft_consensus.cc:686] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:04.473377 7875 consensus_queue.cc:237] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.474772 6152 catalog_manager.cc:5484] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:04.545970 7601 raft_consensus.cc:482] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.546078 7601 raft_consensus.cc:504] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.546459 7601 leader_election.cc:288] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.546751 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc995af1acb9459f9096867bc3b075f4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:04.546823 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc995af1acb9459f9096867bc3b075f4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:04.546891 6428 raft_consensus.cc:2454] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:04.546960 6561 raft_consensus.cc:2454] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:04.547127 6235 leader_election.cc:302] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.547298 7601 raft_consensus.cc:2781] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.547360 7601 raft_consensus.cc:482] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.547389 7601 raft_consensus.cc:3037] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.548220 7601 raft_consensus.cc:504] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.548486 7601 leader_election.cc:288] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:04.548743 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc995af1acb9459f9096867bc3b075f4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:04.548861 6428 raft_consensus.cc:3037] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.548844 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc995af1acb9459f9096867bc3b075f4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:04.548972 6561 raft_consensus.cc:3037] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.549698 6428 raft_consensus.cc:2454] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:04.549918 6561 raft_consensus.cc:2454] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:04.549929 6235 leader_election.cc:302] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:04.550189 7877 raft_consensus.cc:2781] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.550276 7877 raft_consensus.cc:686] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:04.550415 7877 consensus_queue.cc:237] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:04.552306 6153 catalog_manager.cc:5484] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:04.699189 7988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:04.699383 7988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:04.702337 7988 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 07:59:04.777799 6561 raft_consensus.cc:1261] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.778306 7877 consensus_queue.cc:1035] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.781221 6690 raft_consensus.cc:1261] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.781600 6298 raft_consensus.cc:1261] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.781841 7875 consensus_queue.cc:1035] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.782117 6428 raft_consensus.cc:1261] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.782131 7875 consensus_queue.cc:1035] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.783708 6297 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.784677 6560 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:04.784883 7914 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.785182 7798 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.791203 7601 consensus_queue.cc:1035] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:04.807010 7845 mvcc.cc:205] Tried to move back new op lower bound from 7023506000998088704 to 7023505999764049920. Current Snapshot: MvccSnapshot[applied={T|T < 7023506000998088704}]
INSERT report
rows total: 2000
time total: 110.132 ms
time per row: 0.055066 ms
I20240503 07:59:04.856106 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:04.861759 6278 tablet_service.cc:1467] Processing CreateTablet for tablet d4308bc87be24f28b6566ce76cae6e50 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.862033 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4308bc87be24f28b6566ce76cae6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.862552 6409 tablet_service.cc:1467] Processing CreateTablet for tablet d4308bc87be24f28b6566ce76cae6e50 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.862659 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 2a10085c34be4d31aae9f56a2ec1ba23 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.862789 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4308bc87be24f28b6566ce76cae6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.862926 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a10085c34be4d31aae9f56a2ec1ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.862942 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d4308bc87be24f28b6566ce76cae6e50 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.863201 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4308bc87be24f28b6566ce76cae6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.864248 7861 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:04.864791 7863 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:04.865051 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 2a10085c34be4d31aae9f56a2ec1ba23 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.865134 7861 tablet_bootstrap.cc:654] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.865298 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a10085c34be4d31aae9f56a2ec1ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.865414 6671 tablet_service.cc:1467] Processing CreateTablet for tablet ebe60a4754dd40d0964493e58929cebb (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.865576 7863 tablet_bootstrap.cc:654] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.865609 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe60a4754dd40d0964493e58929cebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.866621 6540 tablet_service.cc:1467] Processing CreateTablet for tablet ebe60a4754dd40d0964493e58929cebb (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.866858 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe60a4754dd40d0964493e58929cebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.868368 7861 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:04.868467 7861 ts_tablet_manager.cc:1367] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:04.868618 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 2a10085c34be4d31aae9f56a2ec1ba23 (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.868705 7866 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:04.868850 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a10085c34be4d31aae9f56a2ec1ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.868955 7861 raft_consensus.cc:348] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.869061 7861 raft_consensus.cc:374] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.869093 7861 raft_consensus.cc:729] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.869225 7861 consensus_queue.cc:260] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.869534 7861 ts_tablet_manager.cc:1398] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.869555 7866 tablet_bootstrap.cc:654] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.871081 8032 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:04.871498 7863 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:04.871587 7863 ts_tablet_manager.cc:1367] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:04.872040 8032 tablet_bootstrap.cc:654] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.872027 7863 raft_consensus.cc:348] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.872138 7863 raft_consensus.cc:374] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.872181 7863 raft_consensus.cc:729] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.872309 7863 consensus_queue.cc:260] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.872587 7863 ts_tablet_manager.cc:1398] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.872747 7863 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:04.873502 7863 tablet_bootstrap.cc:654] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.873595 7866 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:04.873682 7866 ts_tablet_manager.cc:1367] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:04.874073 7866 raft_consensus.cc:348] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.874159 7866 raft_consensus.cc:374] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.874186 7866 raft_consensus.cc:729] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.874294 7866 consensus_queue.cc:260] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.874410 6278 tablet_service.cc:1467] Processing CreateTablet for tablet ebe60a4754dd40d0964493e58929cebb (DEFAULT_TABLE table=rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:04.874477 7863 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:04.874543 7863 ts_tablet_manager.cc:1367] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.874585 7866 ts_tablet_manager.cc:1398] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.874653 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe60a4754dd40d0964493e58929cebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:04.874742 7866 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:04.874923 7863 raft_consensus.cc:348] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.875005 7863 raft_consensus.cc:374] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.875031 7863 raft_consensus.cc:729] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.875135 7863 consensus_queue.cc:260] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.875366 7863 ts_tablet_manager.cc:1398] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:04.875572 7866 tablet_bootstrap.cc:654] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.876432 7861 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:04.877096 7866 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:04.877146 8032 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:04.877236 8032 ts_tablet_manager.cc:1367] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:04.877272 7866 ts_tablet_manager.cc:1367] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:04.877375 7861 tablet_bootstrap.cc:654] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.877622 8032 raft_consensus.cc:348] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.877729 8032 raft_consensus.cc:374] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.877764 8032 raft_consensus.cc:729] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.877866 8032 consensus_queue.cc:260] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.878034 7866 raft_consensus.cc:348] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.878132 8032 ts_tablet_manager.cc:1398] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:04.878218 7866 raft_consensus.cc:374] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.878299 8032 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:04.878305 7866 raft_consensus.cc:729] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.878513 7866 consensus_queue.cc:260] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.878732 7861 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:04.878819 7861 ts_tablet_manager.cc:1367] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:04.878993 8032 tablet_bootstrap.cc:654] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.879012 7866 ts_tablet_manager.cc:1398] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:04.879240 7861 raft_consensus.cc:348] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.879338 7861 raft_consensus.cc:374] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.879371 7861 raft_consensus.cc:729] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.879492 7861 consensus_queue.cc:260] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.879782 7861 ts_tablet_manager.cc:1398] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:04.880611 8032 tablet_bootstrap.cc:492] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:04.880683 8032 ts_tablet_manager.cc:1367] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:04.881108 8032 raft_consensus.cc:348] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.881203 8032 raft_consensus.cc:374] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.881233 8032 raft_consensus.cc:729] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.881328 8032 consensus_queue.cc:260] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:04.881611 8032 ts_tablet_manager.cc:1398] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.881763 8032 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:04.882511 8032 tablet_bootstrap.cc:654] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:04.883630 8032 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:04.883713 8032 ts_tablet_manager.cc:1367] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:04.884071 8032 raft_consensus.cc:348] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.884166 8032 raft_consensus.cc:374] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:04.884189 8032 raft_consensus.cc:729] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:04.884284 8032 consensus_queue.cc:260] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.884527 8032 ts_tablet_manager.cc:1398] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:04.934187 7877 raft_consensus.cc:482] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:04.934289 7877 raft_consensus.cc:504] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.934650 7877 leader_election.cc:288] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:04.934907 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4308bc87be24f28b6566ce76cae6e50" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:04.934924 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4308bc87be24f28b6566ce76cae6e50" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:04.935025 6428 raft_consensus.cc:2454] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:04.935024 6690 raft_consensus.cc:2454] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:04.935273 6234 leader_election.cc:302] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:04.935485 7877 raft_consensus.cc:2781] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:04.935540 7877 raft_consensus.cc:482] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:04.935565 7877 raft_consensus.cc:3037] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.936298 7877 raft_consensus.cc:504] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.936596 7877 leader_election.cc:288] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:04.936897 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4308bc87be24f28b6566ce76cae6e50" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:04.936877 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4308bc87be24f28b6566ce76cae6e50" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:04.937002 6428 raft_consensus.cc:3037] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.937004 6690 raft_consensus.cc:3037] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:04.937826 6690 raft_consensus.cc:2454] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:04.937901 6428 raft_consensus.cc:2454] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:04.938078 6234 leader_election.cc:302] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:04.938222 7877 raft_consensus.cc:2781] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:04.938300 7877 raft_consensus.cc:686] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:04.938438 7877 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:04.940104 6153 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:05.043246 7969 raft_consensus.cc:482] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.043381 7969 raft_consensus.cc:504] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.043773 7969 leader_election.cc:288] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.044126 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe60a4754dd40d0964493e58929cebb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:05.044188 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe60a4754dd40d0964493e58929cebb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:05.044250 6297 raft_consensus.cc:2454] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.044456 6561 raft_consensus.cc:2454] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.044591 7904 raft_consensus.cc:482] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.044674 6625 leader_election.cc:302] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.044680 7904 raft_consensus.cc:504] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.044864 7969 raft_consensus.cc:2781] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.044910 7969 raft_consensus.cc:482] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.044934 7969 raft_consensus.cc:3037] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.045049 7904 leader_election.cc:288] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:05.045359 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a10085c34be4d31aae9f56a2ec1ba23" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:05.045491 6690 raft_consensus.cc:2454] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:05.045469 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a10085c34be4d31aae9f56a2ec1ba23" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:05.045583 6561 raft_consensus.cc:2454] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:05.045799 6365 leader_election.cc:302] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.045967 7904 raft_consensus.cc:2781] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.046010 7904 raft_consensus.cc:482] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.046034 7904 raft_consensus.cc:3037] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.046169 7969 raft_consensus.cc:504] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.046499 7969 leader_election.cc:288] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.046833 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe60a4754dd40d0964493e58929cebb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:05.046814 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe60a4754dd40d0964493e58929cebb" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:05.046870 7904 raft_consensus.cc:504] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.046929 6297 raft_consensus.cc:3037] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.046929 6561 raft_consensus.cc:3037] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.047191 7904 leader_election.cc:288] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:05.047549 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a10085c34be4d31aae9f56a2ec1ba23" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:05.047660 6690 raft_consensus.cc:3037] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.047763 6297 raft_consensus.cc:2454] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.048028 6561 raft_consensus.cc:2454] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.048029 6625 leader_election.cc:302] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.048134 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a10085c34be4d31aae9f56a2ec1ba23" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:05.048218 6560 raft_consensus.cc:3037] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.048348 7969 raft_consensus.cc:2781] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.048548 7969 raft_consensus.cc:686] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:05.048689 7969 consensus_queue.cc:237] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.048552 6690 raft_consensus.cc:2454] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:05.048957 6560 raft_consensus.cc:2454] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:05.049165 6365 leader_election.cc:302] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.049340 7904 raft_consensus.cc:2781] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.049429 7904 raft_consensus.cc:686] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:05.049541 7904 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.050489 6153 catalog_manager.cc:5484] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:05.051033 6157 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:05.162914 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:05.163101 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:05.166054 8045 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 07:59:05.242827 6690 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.243232 7601 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.243532 6429 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.244125 7877 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.248637 6297 raft_consensus.cc:1261] T ebe60a4754dd40d0964493e58929cebb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.250053 7837 mvcc.cc:205] Tried to move back new op lower bound from 7023506002910773248 to 7023506001669013504. Current Snapshot: MvccSnapshot[applied={T|T < 7023506002910773248}]
I20240503 07:59:05.250201 6560 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.250258 6561 raft_consensus.cc:1261] T ebe60a4754dd40d0964493e58929cebb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.251250 7904 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.252086 6691 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.254848 7970 mvcc.cc:205] Tried to move back new op lower bound from 7023506002937602048 to 7023506002123874304. Current Snapshot: MvccSnapshot[applied={T|T < 7023506002937602048}]
I20240503 07:59:05.260016 8041 consensus_queue.cc:1035] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.260283 8041 consensus_queue.cc:1035] T ebe60a4754dd40d0964493e58929cebb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.261384 7904 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.271663 7842 mvcc.cc:205] Tried to move back new op lower bound from 7023506002934747136 to 7023506002121019392. Current Snapshot: MvccSnapshot[applied={T|T < 7023506002934747136}]
INSERT report
rows total: 2000
time total: 114.887 ms
time per row: 0.0574433 ms
I20240503 07:59:05.318508 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:05.323647 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.323915 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fbfb84cef74316a77b3ccfab49a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.324033 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.324285 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18677aa766454b8c98a1a2f4a48feee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.324433 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.324653 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fbfb84cef74316a77b3ccfab49a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.325958 7861 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:05.326737 7861 tablet_bootstrap.cc:654] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.326978 6540 tablet_service.cc:1467] Processing CreateTablet for tablet b29dafc7b564493a8fdf064e7d3461ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.326978 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.327219 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29dafc7b564493a8fdf064e7d3461ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.327255 8032 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:05.327329 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18677aa766454b8c98a1a2f4a48feee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.328058 8032 tablet_bootstrap.cc:654] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.329077 7866 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:05.329510 6279 tablet_service.cc:1467] Processing CreateTablet for tablet b29dafc7b564493a8fdf064e7d3461ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.329715 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29dafc7b564493a8fdf064e7d3461ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.329835 7866 tablet_bootstrap.cc:654] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.330763 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.331019 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fbfb84cef74316a77b3ccfab49a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.331806 8032 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:05.331877 8032 ts_tablet_manager.cc:1367] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:05.332250 8032 raft_consensus.cc:348] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.332348 8032 raft_consensus.cc:374] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.332374 8032 raft_consensus.cc:729] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.332466 8032 consensus_queue.cc:260] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.332573 7863 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:05.332695 8032 ts_tablet_manager.cc:1398] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:05.332729 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.333268 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18677aa766454b8c98a1a2f4a48feee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.333356 7863 tablet_bootstrap.cc:654] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.334501 7863 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:05.334574 7863 ts_tablet_manager.cc:1367] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:05.334954 7863 raft_consensus.cc:348] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.335069 7863 raft_consensus.cc:374] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.335103 7863 raft_consensus.cc:729] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.335212 7863 consensus_queue.cc:260] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.335470 7863 ts_tablet_manager.cc:1398] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.335637 7863 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:05.336393 7863 tablet_bootstrap.cc:654] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.337626 6672 tablet_service.cc:1467] Processing CreateTablet for tablet b29dafc7b564493a8fdf064e7d3461ef (DEFAULT_TABLE table=rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.337841 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b29dafc7b564493a8fdf064e7d3461ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.340628 8032 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:05.341300 7861 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:05.341378 7861 ts_tablet_manager.cc:1367] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:05.341392 8032 tablet_bootstrap.cc:654] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.341797 7861 raft_consensus.cc:348] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.341903 7861 raft_consensus.cc:374] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.341935 7861 raft_consensus.cc:729] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.342000 7863 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:05.342063 7863 ts_tablet_manager.cc:1367] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:05.342047 7861 consensus_queue.cc:260] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.342319 7861 ts_tablet_manager.cc:1398] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.342414 7863 raft_consensus.cc:348] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.342478 7861 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:05.342495 7863 raft_consensus.cc:374] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.342522 7863 raft_consensus.cc:729] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.342589 8032 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:05.342654 8032 ts_tablet_manager.cc:1367] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:05.342619 7863 consensus_queue.cc:260] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.342871 7863 ts_tablet_manager.cc:1398] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:05.343021 8032 raft_consensus.cc:348] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.343112 8032 raft_consensus.cc:374] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.343142 8032 raft_consensus.cc:729] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.343238 8032 consensus_queue.cc:260] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.343286 7861 tablet_bootstrap.cc:654] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.343505 8032 ts_tablet_manager.cc:1398] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.344326 7866 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:05.344431 7866 ts_tablet_manager.cc:1367] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240503 07:59:05.344863 7866 raft_consensus.cc:348] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.344974 7866 raft_consensus.cc:374] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.345006 7866 raft_consensus.cc:729] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.345129 7866 consensus_queue.cc:260] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.345432 7866 ts_tablet_manager.cc:1398] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.345582 7866 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:05.346354 7866 tablet_bootstrap.cc:654] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.346560 7861 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:05.346637 7861 ts_tablet_manager.cc:1367] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:05.347018 7861 raft_consensus.cc:348] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.347110 7861 raft_consensus.cc:374] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.347138 7861 raft_consensus.cc:729] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.347259 7861 consensus_queue.cc:260] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.347849 7866 tablet_bootstrap.cc:492] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:05.347959 7866 ts_tablet_manager.cc:1367] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:05.348353 7866 raft_consensus.cc:348] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.348461 7866 raft_consensus.cc:374] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.348469 7861 ts_tablet_manager.cc:1398] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:05.348496 7866 raft_consensus.cc:729] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.348670 7861 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:05.348642 7866 consensus_queue.cc:260] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.348914 7866 ts_tablet_manager.cc:1398] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.350512 7861 tablet_bootstrap.cc:654] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.351459 7861 tablet_bootstrap.cc:492] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:05.351524 7861 ts_tablet_manager.cc:1367] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:05.351820 7861 raft_consensus.cc:348] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.351907 7861 raft_consensus.cc:374] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.351928 7861 raft_consensus.cc:729] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.352234 7861 consensus_queue.cc:260] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.352481 7861 ts_tablet_manager.cc:1398] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.390277 7601 raft_consensus.cc:482] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.390364 7601 raft_consensus.cc:504] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.390684 7601 leader_election.cc:288] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:05.390955 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbfb84cef74316a77b3ccfab49a19b" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:05.390971 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbfb84cef74316a77b3ccfab49a19b" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:05.391083 6691 raft_consensus.cc:2454] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:05.391080 6429 raft_consensus.cc:2454] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:05.391315 6234 leader_election.cc:302] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.391458 7601 raft_consensus.cc:2781] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.391511 7601 raft_consensus.cc:482] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.391553 7601 raft_consensus.cc:3037] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.392304 7601 raft_consensus.cc:504] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.392611 7601 leader_election.cc:288] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:05.392881 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbfb84cef74316a77b3ccfab49a19b" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:05.392997 6691 raft_consensus.cc:3037] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.392951 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbfb84cef74316a77b3ccfab49a19b" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:05.393085 6429 raft_consensus.cc:3037] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.393707 6691 raft_consensus.cc:2454] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:05.393882 6429 raft_consensus.cc:2454] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:05.393950 6234 leader_election.cc:302] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.394088 7601 raft_consensus.cc:2781] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.394165 7601 raft_consensus.cc:686] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:05.394282 7601 consensus_queue.cc:237] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.395444 7601 raft_consensus.cc:482] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.395519 7601 raft_consensus.cc:504] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.395820 7601 leader_election.cc:288] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.396122 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18677aa766454b8c98a1a2f4a48feee2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:05.396168 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18677aa766454b8c98a1a2f4a48feee2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:05.396106 6157 catalog_manager.cc:5484] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:05.396229 6429 raft_consensus.cc:2454] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:05.396289 6561 raft_consensus.cc:2454] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:05.396497 6235 leader_election.cc:302] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.396713 7601 raft_consensus.cc:2781] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.396759 7601 raft_consensus.cc:482] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.396813 7601 raft_consensus.cc:3037] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.397526 7601 raft_consensus.cc:504] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.397811 7601 leader_election.cc:288] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.398195 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18677aa766454b8c98a1a2f4a48feee2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:05.398288 6561 raft_consensus.cc:3037] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.398304 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18677aa766454b8c98a1a2f4a48feee2" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:05.398420 6429 raft_consensus.cc:3037] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.398963 6561 raft_consensus.cc:2454] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:05.399062 6429 raft_consensus.cc:2454] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:05.399225 6232 leader_election.cc:302] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.399371 7601 raft_consensus.cc:2781] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.399456 7601 raft_consensus.cc:686] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:05.399569 7601 consensus_queue.cc:237] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.401160 6157 catalog_manager.cc:5484] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:05.423131 8077 raft_consensus.cc:482] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.423213 8077 raft_consensus.cc:504] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.423522 8077 leader_election.cc:288] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:05.423781 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29dafc7b564493a8fdf064e7d3461ef" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:05.423857 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29dafc7b564493a8fdf064e7d3461ef" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:05.423888 6561 raft_consensus.cc:2454] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.423980 6297 raft_consensus.cc:2454] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.424124 6625 leader_election.cc:302] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:05.424273 8077 raft_consensus.cc:2781] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.424335 8077 raft_consensus.cc:482] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.424379 8077 raft_consensus.cc:3037] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.425326 8077 raft_consensus.cc:504] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.425591 8077 leader_election.cc:288] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:05.425839 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29dafc7b564493a8fdf064e7d3461ef" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:05.425897 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b29dafc7b564493a8fdf064e7d3461ef" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:05.425947 6561 raft_consensus.cc:3037] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.425973 6297 raft_consensus.cc:3037] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.426752 6561 raft_consensus.cc:2454] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.426752 6297 raft_consensus.cc:2454] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.427004 6625 leader_election.cc:302] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:05.427163 8077 raft_consensus.cc:2781] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.427248 8077 raft_consensus.cc:686] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:05.427364 8077 consensus_queue.cc:237] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:05.428901 6157 catalog_manager.cc:5484] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:05.538654 8098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:05.538838 8098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:05.542378 8098 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 07:59:05.632614 6297 raft_consensus.cc:1261] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.632992 6560 raft_consensus.cc:1261] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.633826 6429 raft_consensus.cc:1261] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.634084 6428 raft_consensus.cc:1261] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.634249 8077 consensus_queue.cc:1035] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.634341 6691 raft_consensus.cc:1261] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.634480 7601 consensus_queue.cc:1035] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.634743 7601 consensus_queue.cc:1035] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.634927 6560 raft_consensus.cc:1261] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:05.635668 8077 consensus_queue.cc:1035] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.636832 7877 consensus_queue.cc:1035] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:05.637037 8075 consensus_queue.cc:1035] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.292 ms
time per row: 0.0556459 ms
I20240503 07:59:05.704638 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:05.712955 6409 tablet_service.cc:1467] Processing CreateTablet for tablet c597bc1a19274f55b9d60558db35fcae (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.713253 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c597bc1a19274f55b9d60558db35fcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.713909 6672 tablet_service.cc:1467] Processing CreateTablet for tablet c597bc1a19274f55b9d60558db35fcae (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.713990 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 558c0eb4a4334dbe91ac60fad4e820dc (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.714005 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 8384cbf3f368424d9f8a9060efe9c0f2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.714147 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c597bc1a19274f55b9d60558db35fcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.714171 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c0eb4a4334dbe91ac60fad4e820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.714411 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 8384cbf3f368424d9f8a9060efe9c0f2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.714612 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8384cbf3f368424d9f8a9060efe9c0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.715512 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 558c0eb4a4334dbe91ac60fad4e820dc (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.715706 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c0eb4a4334dbe91ac60fad4e820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.716086 8032 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:05.716626 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8384cbf3f368424d9f8a9060efe9c0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.717155 8032 tablet_bootstrap.cc:654] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.717540 7863 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:05.717574 6539 tablet_service.cc:1467] Processing CreateTablet for tablet c597bc1a19274f55b9d60558db35fcae (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.717586 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 558c0eb4a4334dbe91ac60fad4e820dc (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.717790 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c597bc1a19274f55b9d60558db35fcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.717864 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c0eb4a4334dbe91ac60fad4e820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.718457 7863 tablet_bootstrap.cc:654] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.719130 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 8384cbf3f368424d9f8a9060efe9c0f2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:05.719347 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8384cbf3f368424d9f8a9060efe9c0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:05.719834 7861 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:05.720461 7866 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:05.720674 7861 tablet_bootstrap.cc:654] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.721421 7866 tablet_bootstrap.cc:654] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.722476 7866 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:05.722565 7866 ts_tablet_manager.cc:1367] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:05.722985 7866 raft_consensus.cc:348] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.723109 7866 raft_consensus.cc:374] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.723142 7866 raft_consensus.cc:729] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.723145 7863 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:05.723234 7863 ts_tablet_manager.cc:1367] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:05.723246 7866 consensus_queue.cc:260] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.723528 7866 ts_tablet_manager.cc:1398] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.723728 7863 raft_consensus.cc:348] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.723826 7863 raft_consensus.cc:374] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.723860 7863 raft_consensus.cc:729] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.723976 7863 consensus_queue.cc:260] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.724251 7863 ts_tablet_manager.cc:1398] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.724260 7866 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:05.724421 7863 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:05.725428 7863 tablet_bootstrap.cc:654] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.725611 7866 tablet_bootstrap.cc:654] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.726603 7866 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:05.726676 7866 ts_tablet_manager.cc:1367] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:05.727063 7866 raft_consensus.cc:348] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.727157 7866 raft_consensus.cc:374] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.727187 7866 raft_consensus.cc:729] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.727290 7866 consensus_queue.cc:260] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.727526 7866 ts_tablet_manager.cc:1398] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.733062 7861 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:05.733145 7861 ts_tablet_manager.cc:1367] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 07:59:05.733554 7861 raft_consensus.cc:348] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.733635 7861 raft_consensus.cc:374] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.733654 7861 raft_consensus.cc:729] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.733758 7861 consensus_queue.cc:260] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.733986 7861 ts_tablet_manager.cc:1398] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:05.735917 7863 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:05.736001 7863 ts_tablet_manager.cc:1367] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240503 07:59:05.736410 7863 raft_consensus.cc:348] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.736518 7863 raft_consensus.cc:374] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.736552 7863 raft_consensus.cc:729] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.736672 7863 consensus_queue.cc:260] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.736935 7863 ts_tablet_manager.cc:1398] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.737115 7863 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:05.737829 7863 tablet_bootstrap.cc:654] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.738756 8032 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:05.738849 8032 ts_tablet_manager.cc:1367] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:05.738874 7863 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:05.738948 7863 ts_tablet_manager.cc:1367] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:05.739250 8032 raft_consensus.cc:348] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.739276 7863 raft_consensus.cc:348] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.739352 7863 raft_consensus.cc:374] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.739354 8032 raft_consensus.cc:374] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.739377 7863 raft_consensus.cc:729] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.739385 8032 raft_consensus.cc:729] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.739487 7863 consensus_queue.cc:260] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.739485 8032 consensus_queue.cc:260] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.739710 7863 ts_tablet_manager.cc:1398] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.739727 8032 ts_tablet_manager.cc:1398] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.739892 8032 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:05.740677 8032 tablet_bootstrap.cc:654] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.745325 8032 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:05.745442 8032 ts_tablet_manager.cc:1367] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:05.745800 8032 raft_consensus.cc:348] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.745875 8032 raft_consensus.cc:374] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.745903 8032 raft_consensus.cc:729] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.746011 8032 consensus_queue.cc:260] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.746251 8032 ts_tablet_manager.cc:1398] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:05.746400 8032 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:05.747473 8032 tablet_bootstrap.cc:654] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:05.748849 8032 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:05.748945 8032 ts_tablet_manager.cc:1367] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:05.749275 8032 raft_consensus.cc:348] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.749351 8032 raft_consensus.cc:374] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:05.749372 8032 raft_consensus.cc:729] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:05.749459 8032 consensus_queue.cc:260] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.749678 8032 ts_tablet_manager.cc:1398] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:05.778698 8140 raft_consensus.cc:482] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.778795 8140 raft_consensus.cc:504] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.779177 8140 leader_election.cc:288] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:05.779433 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c597bc1a19274f55b9d60558db35fcae" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:05.779475 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c597bc1a19274f55b9d60558db35fcae" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:05.779568 6429 raft_consensus.cc:2454] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:05.779568 6691 raft_consensus.cc:2454] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:05.779829 6497 leader_election.cc:302] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.779985 8140 raft_consensus.cc:2781] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.780051 8140 raft_consensus.cc:482] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.780093 8140 raft_consensus.cc:3037] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.780925 8140 raft_consensus.cc:504] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.781304 8140 leader_election.cc:288] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:05.781566 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c597bc1a19274f55b9d60558db35fcae" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:05.781602 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c597bc1a19274f55b9d60558db35fcae" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:05.781690 6429 raft_consensus.cc:3037] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.781690 6691 raft_consensus.cc:3037] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.782505 6691 raft_consensus.cc:2454] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:05.782505 6429 raft_consensus.cc:2454] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:05.782778 6497 leader_election.cc:302] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.782961 8140 raft_consensus.cc:2781] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.783066 8140 raft_consensus.cc:686] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:05.783203 8140 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.784765 6153 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:05.809907 7798 raft_consensus.cc:482] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.809984 7798 raft_consensus.cc:504] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.810277 7798 leader_election.cc:288] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.810591 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8384cbf3f368424d9f8a9060efe9c0f2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:05.810626 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8384cbf3f368424d9f8a9060efe9c0f2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:05.810690 6560 raft_consensus.cc:2454] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:05.810747 6691 raft_consensus.cc:2454] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:05.810974 6363 leader_election.cc:302] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.811224 7798 raft_consensus.cc:2781] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.811285 7798 raft_consensus.cc:482] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.811321 7798 raft_consensus.cc:3037] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.812034 7798 raft_consensus.cc:504] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.812295 7798 leader_election.cc:288] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:05.812613 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8384cbf3f368424d9f8a9060efe9c0f2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:05.812606 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8384cbf3f368424d9f8a9060efe9c0f2" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:05.812721 6560 raft_consensus.cc:3037] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.812741 6691 raft_consensus.cc:3037] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.813618 6691 raft_consensus.cc:2454] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:05.813618 6560 raft_consensus.cc:2454] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:05.813961 6363 leader_election.cc:302] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.814108 7798 raft_consensus.cc:2781] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.814179 7798 raft_consensus.cc:686] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:05.814299 7798 consensus_queue.cc:237] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:05.818199 6153 catalog_manager.cc:5484] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:05.903342 8041 raft_consensus.cc:482] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:05.903455 8041 raft_consensus.cc:504] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.903860 8041 leader_election.cc:288] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:05.904157 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c0eb4a4334dbe91ac60fad4e820dc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:05.904167 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c0eb4a4334dbe91ac60fad4e820dc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:05.904278 6429 raft_consensus.cc:2454] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.904289 6297 raft_consensus.cc:2454] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:05.904541 6628 leader_election.cc:302] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.904732 8041 raft_consensus.cc:2781] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:05.904831 8041 raft_consensus.cc:482] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:05.904886 8041 raft_consensus.cc:3037] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.905684 8041 raft_consensus.cc:504] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.906036 8041 leader_election.cc:288] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:05.906301 6297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c0eb4a4334dbe91ac60fad4e820dc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:05.906348 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c0eb4a4334dbe91ac60fad4e820dc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:05.906435 6429 raft_consensus.cc:3037] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.906443 6297 raft_consensus.cc:3037] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:05.907162 6429 raft_consensus.cc:2454] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.907279 6297 raft_consensus.cc:2454] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:05.907451 6628 leader_election.cc:302] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:05.907605 8041 raft_consensus.cc:2781] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:05.907675 8041 raft_consensus.cc:686] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:05.907810 8041 consensus_queue.cc:237] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:05.909821 6153 catalog_manager.cc:5484] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:06.022920 8155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:06.023121 8155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:06.026197 8155 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 07:59:06.117166 6560 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.117929 6691 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.117970 6429 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.118366 8136 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.118647 8136 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.122599 6298 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.122995 8041 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.123116 6429 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.123608 7904 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.124307 8077 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.125509 6690 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.126869 7798 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.451 ms
time per row: 0.0592253 ms
I20240503 07:59:06.181947 6149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:06.187592 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.187795 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.187880 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f58a27f2a4a2593e8219b1d7bd894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.188066 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f58a27f2a4a2593e8219b1d7bd894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.188562 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 065d5d0851e840939802896f9bddb84c (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.188830 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065d5d0851e840939802896f9bddb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.188760 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 065d5d0851e840939802896f9bddb84c (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.189015 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065d5d0851e840939802896f9bddb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.189309 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.189529 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f58a27f2a4a2593e8219b1d7bd894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.190207 7863 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:06.190935 6541 tablet_service.cc:1467] Processing CreateTablet for tablet aa754582dc9e4518965f3a2c1836cbee (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.190950 6277 tablet_service.cc:1467] Processing CreateTablet for tablet aa754582dc9e4518965f3a2c1836cbee (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.191084 7863 tablet_bootstrap.cc:654] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.191129 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa754582dc9e4518965f3a2c1836cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.191151 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa754582dc9e4518965f3a2c1836cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.191617 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 065d5d0851e840939802896f9bddb84c (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.191812 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065d5d0851e840939802896f9bddb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.192121 6410 tablet_service.cc:1467] Processing CreateTablet for tablet aa754582dc9e4518965f3a2c1836cbee (DEFAULT_TABLE table=rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.192312 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa754582dc9e4518965f3a2c1836cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.192903 7866 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:06.193023 8032 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:06.193632 7866 tablet_bootstrap.cc:654] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.193631 7861 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:06.193845 8032 tablet_bootstrap.cc:654] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.194382 7861 tablet_bootstrap.cc:654] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.195222 8032 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:06.195288 8032 ts_tablet_manager.cc:1367] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:06.195655 8032 raft_consensus.cc:348] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.195755 8032 raft_consensus.cc:374] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.195778 8032 raft_consensus.cc:729] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.195869 8032 consensus_queue.cc:260] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.196095 8032 ts_tablet_manager.cc:1398] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.197036 7861 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:06.197105 7861 ts_tablet_manager.cc:1367] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:06.197510 7861 raft_consensus.cc:348] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.197589 7861 raft_consensus.cc:374] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.197613 7861 raft_consensus.cc:729] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.197711 7861 consensus_queue.cc:260] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.198040 7861 ts_tablet_manager.cc:1398] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:06.198204 7861 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:06.198621 7863 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:06.198694 7863 ts_tablet_manager.cc:1367] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:06.199040 7861 tablet_bootstrap.cc:654] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.199095 7863 raft_consensus.cc:348] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.199183 7863 raft_consensus.cc:374] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.199213 7863 raft_consensus.cc:729] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.199314 7863 consensus_queue.cc:260] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.199581 7863 ts_tablet_manager.cc:1398] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.199731 7863 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:06.200014 7861 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:06.200083 7861 ts_tablet_manager.cc:1367] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:06.200528 7863 tablet_bootstrap.cc:654] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.201232 7866 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:06.201313 7866 ts_tablet_manager.cc:1367] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:06.201701 7866 raft_consensus.cc:348] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.201789 7866 raft_consensus.cc:374] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.201824 7866 raft_consensus.cc:729] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.201922 7866 consensus_queue.cc:260] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.202168 7866 ts_tablet_manager.cc:1398] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.202322 7866 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:06.203235 7866 tablet_bootstrap.cc:654] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.203357 7861 raft_consensus.cc:348] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.203469 7861 raft_consensus.cc:374] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.203509 7861 raft_consensus.cc:729] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.203640 7861 consensus_queue.cc:260] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.204088 7861 ts_tablet_manager.cc:1398] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:06.204583 7861 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:06.204875 7863 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:06.204959 7863 ts_tablet_manager.cc:1367] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:06.205346 7863 raft_consensus.cc:348] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.205432 7863 raft_consensus.cc:374] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.205467 7861 tablet_bootstrap.cc:654] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.205472 7863 raft_consensus.cc:729] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.205571 7863 consensus_queue.cc:260] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.205816 7863 ts_tablet_manager.cc:1398] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.206421 7866 tablet_bootstrap.cc:492] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:06.206507 7866 ts_tablet_manager.cc:1367] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:06.207082 7866 raft_consensus.cc:348] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.207170 7866 raft_consensus.cc:374] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.207198 7866 raft_consensus.cc:729] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.207302 7866 consensus_queue.cc:260] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.207633 7866 ts_tablet_manager.cc:1398] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.208050 7866 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:06.208210 7861 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:06.208276 7861 ts_tablet_manager.cc:1367] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:06.208663 7861 raft_consensus.cc:348] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.208750 7861 raft_consensus.cc:374] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.208822 7861 raft_consensus.cc:729] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.208843 7866 tablet_bootstrap.cc:654] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.208914 7861 consensus_queue.cc:260] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.209172 7861 ts_tablet_manager.cc:1398] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.210127 7866 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:06.210201 7866 ts_tablet_manager.cc:1367] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:06.210565 7866 raft_consensus.cc:348] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.210660 7866 raft_consensus.cc:374] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.210691 7866 raft_consensus.cc:729] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.210798 7866 consensus_queue.cc:260] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.211030 7866 ts_tablet_manager.cc:1398] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.217945 7969 raft_consensus.cc:482] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.218026 7969 raft_consensus.cc:504] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.218468 7969 leader_election.cc:288] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:06.218689 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065d5d0851e840939802896f9bddb84c" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:06.218712 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065d5d0851e840939802896f9bddb84c" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:06.218793 6298 raft_consensus.cc:2454] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:06.218806 6560 raft_consensus.cc:2454] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:06.219051 6625 leader_election.cc:302] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.219195 7969 raft_consensus.cc:2781] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.219249 7969 raft_consensus.cc:482] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.219293 7969 raft_consensus.cc:3037] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.219925 7969 raft_consensus.cc:504] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.220208 7969 leader_election.cc:288] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:06.220464 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065d5d0851e840939802896f9bddb84c" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:06.220546 6298 raft_consensus.cc:3037] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.220608 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065d5d0851e840939802896f9bddb84c" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:06.220710 6560 raft_consensus.cc:3037] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.221169 6298 raft_consensus.cc:2454] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:06.221451 6560 raft_consensus.cc:2454] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:06.221448 6625 leader_election.cc:302] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.221616 7969 raft_consensus.cc:2781] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.221693 7969 raft_consensus.cc:686] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:06.221812 7969 consensus_queue.cc:237] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:06.223268 6152 catalog_manager.cc:5484] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:06.375232 8140 raft_consensus.cc:482] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.375339 8140 raft_consensus.cc:504] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.375684 8140 leader_election.cc:288] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.376011 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f58a27f2a4a2593e8219b1d7bd894" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:06.376029 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f58a27f2a4a2593e8219b1d7bd894" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:06.376127 6298 raft_consensus.cc:2454] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.376132 6429 raft_consensus.cc:2454] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.376430 6494 leader_election.cc:302] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.376582 8140 raft_consensus.cc:2781] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.376636 8140 raft_consensus.cc:482] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.376659 8140 raft_consensus.cc:3037] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.377117 8015 raft_consensus.cc:482] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.377218 8015 raft_consensus.cc:504] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.377511 8140 raft_consensus.cc:504] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.377555 8015 leader_election.cc:288] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.377874 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa754582dc9e4518965f3a2c1836cbee" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:06.377887 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa754582dc9e4518965f3a2c1836cbee" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:06.377928 8140 leader_election.cc:288] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.378000 6429 raft_consensus.cc:2454] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.378003 6298 raft_consensus.cc:2454] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.378269 6494 leader_election.cc:302] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.378263 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f58a27f2a4a2593e8219b1d7bd894" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:06.378391 6429 raft_consensus.cc:3037] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.378460 8140 raft_consensus.cc:2781] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.378506 8140 raft_consensus.cc:482] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.378477 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f58a27f2a4a2593e8219b1d7bd894" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:06.378540 8140 raft_consensus.cc:3037] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.378569 6298 raft_consensus.cc:3037] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.379185 6429 raft_consensus.cc:2454] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.379386 6298 raft_consensus.cc:2454] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.379371 8140 raft_consensus.cc:504] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.379470 6497 leader_election.cc:302] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:06.379627 8015 raft_consensus.cc:2781] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.379690 8140 leader_election.cc:288] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.379710 8015 raft_consensus.cc:686] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:06.380121 8015 consensus_queue.cc:237] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.381115 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa754582dc9e4518965f3a2c1836cbee" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:06.381228 6298 raft_consensus.cc:3037] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.381469 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa754582dc9e4518965f3a2c1836cbee" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:06.381656 6429 raft_consensus.cc:3037] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.382102 6298 raft_consensus.cc:2454] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.382365 6494 leader_election.cc:302] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.382506 8015 raft_consensus.cc:2781] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.382615 8015 raft_consensus.cc:686] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:06.382750 8015 consensus_queue.cc:237] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:06.382870 6429 raft_consensus.cc:2454] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.383972 6152 catalog_manager.cc:5484] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:06.387480 6152 catalog_manager.cc:5484] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:06.535163 8209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:06.535389 8209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:06.538450 8209 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 07:59:06.621116 6429 raft_consensus.cc:1261] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.621536 8015 consensus_queue.cc:1035] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.623955 6297 raft_consensus.cc:1261] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.624105 6560 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.623955 6298 raft_consensus.cc:1261] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.624507 7969 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.624678 8015 consensus_queue.cc:1035] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.624954 8015 consensus_queue.cc:1035] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.625501 6298 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.625835 7969 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.628016 6429 raft_consensus.cc:1261] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.628549 8015 consensus_queue.cc:1035] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.43 ms
time per row: 0.0567151 ms
I20240503 07:59:06.691597 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:06.697094 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 7f417c61594348428ee9514ded49ee88 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.697391 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f417c61594348428ee9514ded49ee88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.697465 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 7f417c61594348428ee9514ded49ee88 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.697774 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f417c61594348428ee9514ded49ee88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.698375 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.698611 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699571 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.699796 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.699892 8245 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:06.700109 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.700330 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.700800 7863 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:06.700893 8245 tablet_bootstrap.cc:654] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.701613 7863 tablet_bootstrap.cc:654] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.702224 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 39b01318dc1543cba2cc991e9db91c28 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.702443 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b01318dc1543cba2cc991e9db91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.702507 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 39b01318dc1543cba2cc991e9db91c28 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.702754 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b01318dc1543cba2cc991e9db91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.703032 8245 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:06.703152 8245 ts_tablet_manager.cc:1367] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:06.703763 8245 raft_consensus.cc:348] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.703892 8245 raft_consensus.cc:374] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.703927 8245 raft_consensus.cc:729] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.704093 8245 consensus_queue.cc:260] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.704401 7861 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:06.704427 8245 ts_tablet_manager.cc:1398] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:06.704538 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 7f417c61594348428ee9514ded49ee88 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.704599 8245 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:06.704756 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f417c61594348428ee9514ded49ee88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.704826 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 39b01318dc1543cba2cc991e9db91c28 (DEFAULT_TABLE table=rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:06.705032 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b01318dc1543cba2cc991e9db91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:06.705227 7861 tablet_bootstrap.cc:654] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.705446 8245 tablet_bootstrap.cc:654] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.706650 7866 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:06.707345 7861 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:06.707419 7861 ts_tablet_manager.cc:1367] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:06.707522 7866 tablet_bootstrap.cc:654] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.707769 7861 raft_consensus.cc:348] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.707857 7861 raft_consensus.cc:374] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.707886 7861 raft_consensus.cc:729] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.707998 7861 consensus_queue.cc:260] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.708249 7861 ts_tablet_manager.cc:1398] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.708384 7861 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:06.708431 7863 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:06.708523 7863 ts_tablet_manager.cc:1367] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:06.708941 7863 raft_consensus.cc:348] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.709033 7863 raft_consensus.cc:374] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.709066 7863 raft_consensus.cc:729] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.709143 7861 tablet_bootstrap.cc:654] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.709192 7863 consensus_queue.cc:260] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.709477 7863 ts_tablet_manager.cc:1398] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.709892 7866 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:06.709972 7866 ts_tablet_manager.cc:1367] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:06.710348 7866 raft_consensus.cc:348] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.710446 7866 raft_consensus.cc:374] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.710477 7866 raft_consensus.cc:729] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.710598 7866 consensus_queue.cc:260] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.710920 7866 ts_tablet_manager.cc:1398] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:06.711097 7866 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:06.711196 7861 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:06.711261 7861 ts_tablet_manager.cc:1367] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:06.711637 7861 raft_consensus.cc:348] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.711733 7861 raft_consensus.cc:374] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.711772 7861 raft_consensus.cc:729] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.711925 7866 tablet_bootstrap.cc:654] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.711896 7861 consensus_queue.cc:260] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.711998 8075 raft_consensus.cc:482] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.712078 8075 raft_consensus.cc:504] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.712152 7861 ts_tablet_manager.cc:1398] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.712512 8075 leader_election.cc:288] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.713227 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f417c61594348428ee9514ded49ee88" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:06.713349 6690 raft_consensus.cc:2454] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:06.713610 6234 leader_election.cc:302] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.713768 8075 raft_consensus.cc:2781] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.713814 8075 raft_consensus.cc:482] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.713848 8075 raft_consensus.cc:3037] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.713855 7863 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:06.714637 8075 raft_consensus.cc:504] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.714717 7863 tablet_bootstrap.cc:654] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.714928 8075 leader_election.cc:288] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:06.715131 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f417c61594348428ee9514ded49ee88" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:06.715243 6428 raft_consensus.cc:2454] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:06.715510 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f417c61594348428ee9514ded49ee88" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:06.715608 6430 raft_consensus.cc:3037] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.715984 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f417c61594348428ee9514ded49ee88" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:06.716080 6690 raft_consensus.cc:3037] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.716845 6690 raft_consensus.cc:2454] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:06.717145 6234 leader_election.cc:302] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.717291 8075 raft_consensus.cc:2781] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.717394 8075 raft_consensus.cc:686] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:06.717514 8075 consensus_queue.cc:237] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.717679 6430 raft_consensus.cc:2454] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:06.718319 7863 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:06.718400 7863 ts_tablet_manager.cc:1367] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:06.718811 7863 raft_consensus.cc:348] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.718921 7863 raft_consensus.cc:374] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.718964 7863 raft_consensus.cc:729] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.719094 7863 consensus_queue.cc:260] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.719385 7863 ts_tablet_manager.cc:1398] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.719496 6156 catalog_manager.cc:5484] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:06.719851 8245 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:06.719928 8245 ts_tablet_manager.cc:1367] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:06.720397 8245 raft_consensus.cc:348] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.720496 8245 raft_consensus.cc:374] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.720531 8245 raft_consensus.cc:729] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.720552 7866 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:06.720621 7866 ts_tablet_manager.cc:1367] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 07:59:06.720644 8245 consensus_queue.cc:260] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.721069 7866 raft_consensus.cc:348] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.721168 7866 raft_consensus.cc:374] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.721199 7866 raft_consensus.cc:729] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.721299 7866 consensus_queue.cc:260] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.721544 7866 ts_tablet_manager.cc:1398] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:06.722175 8245 ts_tablet_manager.cc:1398] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:06.722363 8245 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:06.723304 8245 tablet_bootstrap.cc:654] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:06.725891 8245 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:06.725977 8245 ts_tablet_manager.cc:1367] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:06.726389 8245 raft_consensus.cc:348] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.726503 8245 raft_consensus.cc:374] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:06.726554 8245 raft_consensus.cc:729] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:06.726675 8245 consensus_queue.cc:260] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.726944 8245 ts_tablet_manager.cc:1398] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:06.742388 8015 raft_consensus.cc:482] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.742476 8015 raft_consensus.cc:504] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.742830 8015 leader_election.cc:288] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:06.743084 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9870eac6ad47d4a9ad5694dcaa1b1f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:06.743139 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9870eac6ad47d4a9ad5694dcaa1b1f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:06.743232 6690 raft_consensus.cc:2454] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.743218 6430 raft_consensus.cc:2454] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:06.743482 6497 leader_election.cc:302] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:06.743625 8015 raft_consensus.cc:2781] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.743680 8015 raft_consensus.cc:482] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.743721 8015 raft_consensus.cc:3037] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.744603 8015 raft_consensus.cc:504] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.744964 8015 leader_election.cc:288] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:06.745244 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9870eac6ad47d4a9ad5694dcaa1b1f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:06.745262 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9870eac6ad47d4a9ad5694dcaa1b1f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:06.745342 6430 raft_consensus.cc:3037] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.745410 6690 raft_consensus.cc:3037] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.746074 6430 raft_consensus.cc:2454] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.746193 6690 raft_consensus.cc:2454] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:06.746380 6497 leader_election.cc:302] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:06.746517 8015 raft_consensus.cc:2781] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.746613 8015 raft_consensus.cc:686] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:06.746733 8015 consensus_queue.cc:237] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:06.748515 6157 catalog_manager.cc:5484] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:06.751690 8041 raft_consensus.cc:482] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:06.751765 8041 raft_consensus.cc:504] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.752038 8041 leader_election.cc:288] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:06.752303 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b01318dc1543cba2cc991e9db91c28" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:06.752396 6560 raft_consensus.cc:2454] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:06.752364 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b01318dc1543cba2cc991e9db91c28" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:06.752486 6298 raft_consensus.cc:2454] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:06.752619 6625 leader_election.cc:302] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:06.752807 8041 raft_consensus.cc:2781] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:06.752874 8041 raft_consensus.cc:482] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:06.752914 8041 raft_consensus.cc:3037] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.753551 8041 raft_consensus.cc:504] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.753809 8041 leader_election.cc:288] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:06.754093 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b01318dc1543cba2cc991e9db91c28" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:06.754127 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b01318dc1543cba2cc991e9db91c28" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:06.754209 6560 raft_consensus.cc:3037] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.754215 6298 raft_consensus.cc:3037] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:06.754848 6298 raft_consensus.cc:2454] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:06.754891 6560 raft_consensus.cc:2454] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:06.755115 6625 leader_election.cc:302] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:06.755380 8041 raft_consensus.cc:2781] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:06.755468 8041 raft_consensus.cc:686] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:06.755584 8041 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:06.757117 6157 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:06.868345 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:06.868543 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:06.871466 8258 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 07:59:06.962474 6561 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.963686 6298 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.964496 7969 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.966810 6430 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.967306 8141 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.969125 6430 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.969214 6690 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.969214 7969 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.969601 8141 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.969669 8075 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.973529 6691 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:06.974463 8075 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:06.989558 7845 mvcc.cc:205] Tried to move back new op lower bound from 7023506009950400512 to 7023506009111953408. Current Snapshot: MvccSnapshot[applied={T|T < 7023506009950400512}]
INSERT report
rows total: 2000
time total: 135.305 ms
time per row: 0.0676527 ms
I20240503 07:59:07.047029 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:07.052551 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.052906 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ce52b9ae543cca72fb92efafc3c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.052954 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 01118afa143f48fabe979e86d0b10c5f (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.053231 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01118afa143f48fabe979e86d0b10c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.053840 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 01118afa143f48fabe979e86d0b10c5f (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.054109 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01118afa143f48fabe979e86d0b10c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.054916 7863 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.055114 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.055260 7861 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.055315 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ce52b9ae543cca72fb92efafc3c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.055805 7863 tablet_bootstrap.cc:654] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.056181 7861 tablet_bootstrap.cc:654] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.056895 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d804e4a97c7d4238944fd03721b4f180 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.057034 7863 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.057111 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d804e4a97c7d4238944fd03721b4f180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.057106 7863 ts_tablet_manager.cc:1367] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.057384 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.057502 7863 raft_consensus.cc:348] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.057597 7863 raft_consensus.cc:374] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.057601 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ce52b9ae543cca72fb92efafc3c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.057626 7863 raft_consensus.cc:729] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.057749 7863 consensus_queue.cc:260] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.058032 7863 ts_tablet_manager.cc:1398] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.058169 7863 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.058660 7866 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.058852 6277 tablet_service.cc:1467] Processing CreateTablet for tablet d804e4a97c7d4238944fd03721b4f180 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.058954 7863 tablet_bootstrap.cc:654] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.059078 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d804e4a97c7d4238944fd03721b4f180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.059440 7866 tablet_bootstrap.cc:654] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.061029 7863 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.061025 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 01118afa143f48fabe979e86d0b10c5f (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.061079 6670 tablet_service.cc:1467] Processing CreateTablet for tablet d804e4a97c7d4238944fd03721b4f180 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.061115 7863 ts_tablet_manager.cc:1367] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:07.061305 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d804e4a97c7d4238944fd03721b4f180. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.061715 7866 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.061785 7866 ts_tablet_manager.cc:1367] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:07.061741 7863 raft_consensus.cc:348] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.061811 7861 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.061831 7863 raft_consensus.cc:374] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.061859 7863 raft_consensus.cc:729] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.061879 7861 ts_tablet_manager.cc:1367] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:07.061980 7863 consensus_queue.cc:260] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.062223 7863 ts_tablet_manager.cc:1398] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.062177 7866 raft_consensus.cc:348] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.062266 7866 raft_consensus.cc:374] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.062295 7866 raft_consensus.cc:729] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.062288 7861 raft_consensus.cc:348] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.062386 7861 raft_consensus.cc:374] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.062417 7861 raft_consensus.cc:729] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.062415 7866 consensus_queue.cc:260] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.062525 7861 consensus_queue.cc:260] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.062682 7866 ts_tablet_manager.cc:1398] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.062796 7861 ts_tablet_manager.cc:1398] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.062827 7866 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.062940 7861 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.063314 8245 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.063594 7866 tablet_bootstrap.cc:654] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.063674 7861 tablet_bootstrap.cc:654] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.064095 8245 tablet_bootstrap.cc:654] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.064808 7861 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.064813 7866 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.064878 7866 ts_tablet_manager.cc:1367] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.064884 7861 ts_tablet_manager.cc:1367] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.065246 7861 raft_consensus.cc:348] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.065272 7866 raft_consensus.cc:348] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.065337 7861 raft_consensus.cc:374] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.065356 7866 raft_consensus.cc:374] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.065367 7861 raft_consensus.cc:729] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.065384 7866 raft_consensus.cc:729] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.065481 7861 consensus_queue.cc:260] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.065521 7866 consensus_queue.cc:260] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.065729 7861 ts_tablet_manager.cc:1398] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.065907 7866 ts_tablet_manager.cc:1398] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.065959 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01118afa143f48fabe979e86d0b10c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.070955 7861 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.071458 8245 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.071518 8245 ts_tablet_manager.cc:1367] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 07:59:07.071669 7861 tablet_bootstrap.cc:654] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.071803 8245 raft_consensus.cc:348] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.071879 8245 raft_consensus.cc:374] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.071897 8245 raft_consensus.cc:729] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.071992 8245 consensus_queue.cc:260] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.072219 8245 ts_tablet_manager.cc:1398] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.072364 8245 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.073174 7861 tablet_bootstrap.cc:492] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.073230 8245 tablet_bootstrap.cc:654] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.073266 7861 ts_tablet_manager.cc:1367] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.073700 7861 raft_consensus.cc:348] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.073810 7861 raft_consensus.cc:374] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.073849 7861 raft_consensus.cc:729] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.073959 7861 consensus_queue.cc:260] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.074213 7861 ts_tablet_manager.cc:1398] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.074563 8245 tablet_bootstrap.cc:492] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.074630 8245 ts_tablet_manager.cc:1367] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:07.075042 8245 raft_consensus.cc:348] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.075121 8245 raft_consensus.cc:374] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.075147 8245 raft_consensus.cc:729] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.075259 8245 consensus_queue.cc:260] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.075526 8245 ts_tablet_manager.cc:1398] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:07.080235 8296 raft_consensus.cc:482] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.080320 8296 raft_consensus.cc:504] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.080646 8296 leader_election.cc:288] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.080958 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d804e4a97c7d4238944fd03721b4f180" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.081091 6298 raft_consensus.cc:2454] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:07.081038 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d804e4a97c7d4238944fd03721b4f180" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:07.081190 6561 raft_consensus.cc:2454] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:07.081622 6625 leader_election.cc:302] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:07.081869 8296 raft_consensus.cc:2781] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.081933 8296 raft_consensus.cc:482] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.081979 8296 raft_consensus.cc:3037] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.082850 8296 raft_consensus.cc:504] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.083133 8296 leader_election.cc:288] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.083482 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d804e4a97c7d4238944fd03721b4f180" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:07.083536 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d804e4a97c7d4238944fd03721b4f180" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.083606 6561 raft_consensus.cc:3037] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.083636 6298 raft_consensus.cc:3037] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.084542 6298 raft_consensus.cc:2454] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:07.084542 6561 raft_consensus.cc:2454] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:07.084861 6625 leader_election.cc:302] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:07.085306 7969 raft_consensus.cc:2781] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.085436 7969 raft_consensus.cc:686] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:07.085568 7969 consensus_queue.cc:237] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.087749 6157 catalog_manager.cc:5484] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:07.145550 8296 raft_consensus.cc:482] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.145687 8296 raft_consensus.cc:504] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.146194 8296 leader_election.cc:288] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:07.146555 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01118afa143f48fabe979e86d0b10c5f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.146608 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01118afa143f48fabe979e86d0b10c5f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:07.146700 6298 raft_consensus.cc:2454] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:07.146734 6429 raft_consensus.cc:2454] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:07.146978 6625 leader_election.cc:302] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:07.147194 8296 raft_consensus.cc:2781] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.147243 8296 raft_consensus.cc:482] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.147274 8296 raft_consensus.cc:3037] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.148238 8296 raft_consensus.cc:504] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.148952 8296 leader_election.cc:288] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:07.149163 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01118afa143f48fabe979e86d0b10c5f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.149360 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01118afa143f48fabe979e86d0b10c5f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:07.149463 6429 raft_consensus.cc:3037] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.150312 6429 raft_consensus.cc:2454] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:07.150557 6298 raft_consensus.cc:3037] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.150581 6628 leader_election.cc:302] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.150776 8296 raft_consensus.cc:2781] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.150861 8296 raft_consensus.cc:686] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:07.151053 8296 consensus_queue.cc:237] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.151356 6298 raft_consensus.cc:2454] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:07.153271 6157 catalog_manager.cc:5484] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:07.201481 8136 raft_consensus.cc:482] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.201577 8136 raft_consensus.cc:504] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.201897 8136 leader_election.cc:288] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:07.202234 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ce52b9ae543cca72fb92efafc3c4d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.202260 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ce52b9ae543cca72fb92efafc3c4d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:07.202359 6298 raft_consensus.cc:2454] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.202420 6429 raft_consensus.cc:2454] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.202628 6494 leader_election.cc:302] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:07.202801 8136 raft_consensus.cc:2781] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.202874 8136 raft_consensus.cc:482] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.202911 8136 raft_consensus.cc:3037] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.203652 8136 raft_consensus.cc:504] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.203936 8136 leader_election.cc:288] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:07.204210 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ce52b9ae543cca72fb92efafc3c4d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:07.204231 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e9ce52b9ae543cca72fb92efafc3c4d" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.204300 6429 raft_consensus.cc:3037] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.204320 6298 raft_consensus.cc:3037] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.205009 6429 raft_consensus.cc:2454] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.205049 6298 raft_consensus.cc:2454] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.205286 6497 leader_election.cc:302] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.205459 8136 raft_consensus.cc:2781] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.205646 8136 raft_consensus.cc:686] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:07.205803 8136 consensus_queue.cc:237] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.207834 6157 catalog_manager.cc:5484] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:07.326117 8311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:07.326313 8311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:07.329682 8311 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 07:59:07.411118 6429 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.411605 8136 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.412988 6298 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.413533 6297 raft_consensus.cc:1261] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.413936 6430 raft_consensus.cc:1261] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.414098 8041 consensus_queue.cc:1035] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.414371 8041 consensus_queue.cc:1035] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.416049 8136 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.422564 6561 raft_consensus.cc:1261] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.422945 7969 consensus_queue.cc:1035] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.428925 6297 raft_consensus.cc:1261] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.429263 8296 consensus_queue.cc:1035] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.430336 8023 mvcc.cc:205] Tried to move back new op lower bound from 7023506011791441920 to 7023506010956263424. Current Snapshot: MvccSnapshot[applied={T|T < 7023506011791441920}]
I20240503 07:59:07.436821 7970 mvcc.cc:205] Tried to move back new op lower bound from 7023506011800408064 to 7023506010732322816. Current Snapshot: MvccSnapshot[applied={T|T < 7023506011800408064}]
I20240503 07:59:07.446455 8293 mvcc.cc:205] Tried to move back new op lower bound from 7023506011838885888 to 7023506010464145408. Current Snapshot: MvccSnapshot[applied={T|T < 7023506011838885888 or (T in {7023506011838885888})}]
INSERT report
rows total: 2000
time total: 103.255 ms
time per row: 0.0516277 ms
I20240503 07:59:07.474200 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:07.478962 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 824c42be8152494fad2cbc76ece56981 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.479161 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 824c42be8152494fad2cbc76ece56981 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.479256 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c42be8152494fad2cbc76ece56981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.479430 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c42be8152494fad2cbc76ece56981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.479614 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 60f643dce3eb4c669b36b0714a31651b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.479750 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 93eb2db48ef449a486a0347a76a8a5ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.479825 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f643dce3eb4c669b36b0714a31651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.479964 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93eb2db48ef449a486a0347a76a8a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.481496 7866 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.481786 7863 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.481824 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 824c42be8152494fad2cbc76ece56981 (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.482045 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c42be8152494fad2cbc76ece56981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.482263 7866 tablet_bootstrap.cc:654] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.482601 7863 tablet_bootstrap.cc:654] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.483564 7866 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.483629 7866 ts_tablet_manager.cc:1367] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.483768 7863 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.483840 7863 ts_tablet_manager.cc:1367] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.484030 7866 raft_consensus.cc:348] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.484124 7866 raft_consensus.cc:374] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.484174 7866 raft_consensus.cc:729] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.484234 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 60f643dce3eb4c669b36b0714a31651b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.484272 7863 raft_consensus.cc:348] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.484282 7866 consensus_queue.cc:260] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.484354 7863 raft_consensus.cc:374] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.484382 7863 raft_consensus.cc:729] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.484452 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f643dce3eb4c669b36b0714a31651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.484490 7863 consensus_queue.cc:260] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.484530 7866 ts_tablet_manager.cc:1398] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.484658 7866 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.484764 7863 ts_tablet_manager.cc:1398] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.485548 7866 tablet_bootstrap.cc:654] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.486016 8245 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.486202 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 93eb2db48ef449a486a0347a76a8a5ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.486378 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93eb2db48ef449a486a0347a76a8a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.486447 7866 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.486532 7866 ts_tablet_manager.cc:1367] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.486768 8245 tablet_bootstrap.cc:654] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.486945 7866 raft_consensus.cc:348] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.487031 7866 raft_consensus.cc:374] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.487059 7866 raft_consensus.cc:729] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.487169 7866 consensus_queue.cc:260] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.487638 7866 ts_tablet_manager.cc:1398] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.487783 7866 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.487792 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 60f643dce3eb4c669b36b0714a31651b (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.488015 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f643dce3eb4c669b36b0714a31651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.488706 7866 tablet_bootstrap.cc:654] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.489323 7863 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.490000 7863 tablet_bootstrap.cc:654] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.491472 8245 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.491559 8245 ts_tablet_manager.cc:1367] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:07.491899 8245 raft_consensus.cc:348] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.491982 8245 raft_consensus.cc:374] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.492010 8245 raft_consensus.cc:729] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.492126 8245 consensus_queue.cc:260] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.492377 8245 ts_tablet_manager.cc:1398] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.492535 8245 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.493206 8245 tablet_bootstrap.cc:654] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.493855 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 93eb2db48ef449a486a0347a76a8a5ea (DEFAULT_TABLE table=rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.493968 7863 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.494036 7863 ts_tablet_manager.cc:1367] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:07.494064 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93eb2db48ef449a486a0347a76a8a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.494331 7863 raft_consensus.cc:348] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.494379 8245 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.494403 7863 raft_consensus.cc:374] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.494423 7863 raft_consensus.cc:729] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.494442 8245 ts_tablet_manager.cc:1367] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.494508 7863 consensus_queue.cc:260] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.494745 7863 ts_tablet_manager.cc:1398] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.494805 8245 raft_consensus.cc:348] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.494892 8245 raft_consensus.cc:374] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.494921 8245 raft_consensus.cc:729] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.495020 8245 consensus_queue.cc:260] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.495270 8245 ts_tablet_manager.cc:1398] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.495815 7863 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.496299 7861 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.496734 7863 tablet_bootstrap.cc:654] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.497108 7861 tablet_bootstrap.cc:654] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.498440 7866 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.498520 7866 ts_tablet_manager.cc:1367] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 07:59:07.498859 7866 raft_consensus.cc:348] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.498939 7866 raft_consensus.cc:374] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.498966 7866 raft_consensus.cc:729] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.499066 7861 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.499063 7866 consensus_queue.cc:260] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.499161 7861 ts_tablet_manager.cc:1367] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:07.499565 7863 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.499579 7861 raft_consensus.cc:348] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.499627 7863 ts_tablet_manager.cc:1367] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:07.499658 7861 raft_consensus.cc:374] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.499681 7861 raft_consensus.cc:729] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.499781 7861 consensus_queue.cc:260] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.499971 7863 raft_consensus.cc:348] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.500062 7863 raft_consensus.cc:374] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.500061 7861 ts_tablet_manager.cc:1398] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.500092 7863 raft_consensus.cc:729] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.500191 7863 consensus_queue.cc:260] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.500479 7863 ts_tablet_manager.cc:1398] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:07.501013 7866 ts_tablet_manager.cc:1398] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.546557 7798 raft_consensus.cc:482] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.546644 7798 raft_consensus.cc:504] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.546962 7798 leader_election.cc:288] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.547230 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f643dce3eb4c669b36b0714a31651b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:07.547250 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f643dce3eb4c669b36b0714a31651b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:07.547341 6561 raft_consensus.cc:2454] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.547346 6690 raft_consensus.cc:2454] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.547590 6363 leader_election.cc:302] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.547760 7798 raft_consensus.cc:2781] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.547816 7798 raft_consensus.cc:482] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.547853 7798 raft_consensus.cc:3037] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.548719 7798 raft_consensus.cc:504] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.549015 7798 leader_election.cc:288] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.549274 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f643dce3eb4c669b36b0714a31651b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:07.549276 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f643dce3eb4c669b36b0714a31651b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:07.549358 6561 raft_consensus.cc:3037] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.549363 6690 raft_consensus.cc:3037] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.550020 6561 raft_consensus.cc:2454] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.550042 6690 raft_consensus.cc:2454] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.550287 6365 leader_election.cc:302] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.550457 7798 raft_consensus.cc:2781] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.550534 7798 raft_consensus.cc:686] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:07.550657 7798 consensus_queue.cc:237] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.552254 6150 catalog_manager.cc:5484] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:07.579316 7798 raft_consensus.cc:482] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.579396 7798 raft_consensus.cc:504] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.579655 7798 leader_election.cc:288] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.579995 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93eb2db48ef449a486a0347a76a8a5ea" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:07.579998 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93eb2db48ef449a486a0347a76a8a5ea" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.580102 6296 raft_consensus.cc:2454] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.580104 6561 raft_consensus.cc:2454] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.580327 6363 leader_election.cc:302] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.580487 7798 raft_consensus.cc:2781] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.580535 7798 raft_consensus.cc:482] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.580578 7798 raft_consensus.cc:3037] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.581281 7798 raft_consensus.cc:504] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.581568 7798 leader_election.cc:288] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:07.581816 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93eb2db48ef449a486a0347a76a8a5ea" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.581848 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93eb2db48ef449a486a0347a76a8a5ea" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:07.581920 6561 raft_consensus.cc:3037] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.581916 6296 raft_consensus.cc:3037] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.582544 6561 raft_consensus.cc:2454] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.582643 6296 raft_consensus.cc:2454] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.582783 6363 leader_election.cc:302] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.582921 7798 raft_consensus.cc:2781] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.583002 7798 raft_consensus.cc:686] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:07.583112 7798 consensus_queue.cc:237] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.584648 6150 catalog_manager.cc:5484] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:07.604987 8015 raft_consensus.cc:482] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.605108 8015 raft_consensus.cc:504] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.605533 8015 leader_election.cc:288] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:07.605769 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c42be8152494fad2cbc76ece56981" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:07.605897 6429 raft_consensus.cc:2454] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.605954 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c42be8152494fad2cbc76ece56981" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:07.606074 6690 raft_consensus.cc:2454] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.606186 6497 leader_election.cc:302] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.606335 8015 raft_consensus.cc:2781] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.606374 8015 raft_consensus.cc:482] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.606405 8015 raft_consensus.cc:3037] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.607283 8015 raft_consensus.cc:504] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.607594 8015 leader_election.cc:288] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:07.607883 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c42be8152494fad2cbc76ece56981" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:07.608000 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c42be8152494fad2cbc76ece56981" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:07.608008 6690 raft_consensus.cc:3037] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.608098 6429 raft_consensus.cc:3037] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.608980 6429 raft_consensus.cc:2454] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.608980 6690 raft_consensus.cc:2454] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.609269 6496 leader_election.cc:302] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:07.609438 8015 raft_consensus.cc:2781] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.609539 8015 raft_consensus.cc:686] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:07.609684 8015 consensus_queue.cc:237] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:07.611713 6150 catalog_manager.cc:5484] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:07.720872 8356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:07.721112 8356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:07.724048 8356 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 07:59:07.803635 6429 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.804018 8015 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.806777 6690 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.808835 6561 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.809864 6296 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.810192 8140 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.810379 6560 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.810577 7798 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.813371 7798 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.813635 7798 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:07.821486 6691 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:07.821820 7798 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.186 ms
time per row: 0.0645929 ms
I20240503 07:59:07.893409 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:07.898376 6670 tablet_service.cc:1467] Processing CreateTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.898505 6410 tablet_service.cc:1467] Processing CreateTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.898697 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e0f5369e8349eba7188b39ae6bac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.898768 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e0f5369e8349eba7188b39ae6bac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.899509 6277 tablet_service.cc:1467] Processing CreateTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.899771 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e0f5369e8349eba7188b39ae6bac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.900578 8245 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.900761 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 7d855e5edf05471983dfdbd40f37560f (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.900991 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d855e5edf05471983dfdbd40f37560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.901510 8245 tablet_bootstrap.cc:654] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.901736 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 77cc5703d29445f691675b12fc1ce191 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.901955 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cc5703d29445f691675b12fc1ce191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.902155 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 7d855e5edf05471983dfdbd40f37560f (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.902413 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d855e5edf05471983dfdbd40f37560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.902729 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 77cc5703d29445f691675b12fc1ce191 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.902884 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cc5703d29445f691675b12fc1ce191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.903808 7861 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.904182 7863 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.904541 7866 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:07.904698 7861 tablet_bootstrap.cc:654] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.904954 7863 tablet_bootstrap.cc:654] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.905331 7866 tablet_bootstrap.cc:654] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.906005 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 7d855e5edf05471983dfdbd40f37560f (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.906031 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 77cc5703d29445f691675b12fc1ce191 (DEFAULT_TABLE table=rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:07.906183 7861 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.906258 7861 ts_tablet_manager.cc:1367] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:07.906270 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d855e5edf05471983dfdbd40f37560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.906706 7861 raft_consensus.cc:348] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.906822 7861 raft_consensus.cc:374] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.906855 7861 raft_consensus.cc:729] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.906859 7866 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:07.906927 7866 ts_tablet_manager.cc:1367] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.906992 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77cc5703d29445f691675b12fc1ce191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:07.906976 7861 consensus_queue.cc:260] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.907285 7861 ts_tablet_manager.cc:1398] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.907357 7866 raft_consensus.cc:348] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.907452 7866 raft_consensus.cc:374] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.907477 7866 raft_consensus.cc:729] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.907478 7861 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.907567 7866 consensus_queue.cc:260] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.907825 7866 ts_tablet_manager.cc:1398] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.908315 7861 tablet_bootstrap.cc:654] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.908653 7863 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.908725 7863 ts_tablet_manager.cc:1367] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:07.909126 7863 raft_consensus.cc:348] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.909214 7863 raft_consensus.cc:374] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.909243 7863 raft_consensus.cc:729] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.909344 7863 consensus_queue.cc:260] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.909608 7863 ts_tablet_manager.cc:1398] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.909765 7863 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:07.910445 7863 tablet_bootstrap.cc:654] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.911887 7863 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:07.911973 7863 ts_tablet_manager.cc:1367] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:07.912082 8245 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.912155 8245 ts_tablet_manager.cc:1367] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:07.912381 7863 raft_consensus.cc:348] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.912475 7863 raft_consensus.cc:374] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.912508 7863 raft_consensus.cc:729] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.912629 8245 raft_consensus.cc:348] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.912724 8245 raft_consensus.cc:374] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.912811 7863 consensus_queue.cc:260] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.913105 7863 ts_tablet_manager.cc:1398] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.912757 8245 raft_consensus.cc:729] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.913584 8245 consensus_queue.cc:260] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:07.913873 8245 ts_tablet_manager.cc:1398] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:07.914052 8245 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.914711 7861 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.914779 7861 ts_tablet_manager.cc:1367] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:07.914861 8245 tablet_bootstrap.cc:654] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.915174 7861 raft_consensus.cc:348] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.915266 7861 raft_consensus.cc:374] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.915297 7861 raft_consensus.cc:729] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.915396 7861 consensus_queue.cc:260] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.915639 7861 ts_tablet_manager.cc:1398] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.915743 8245 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.915781 7861 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:07.915853 8245 ts_tablet_manager.cc:1367] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:07.916389 8245 raft_consensus.cc:348] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.916478 8245 raft_consensus.cc:374] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.916512 8245 raft_consensus.cc:729] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.916636 8245 consensus_queue.cc:260] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.916844 7861 tablet_bootstrap.cc:654] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.916960 8245 ts_tablet_manager.cc:1398] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:07.917129 8245 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:07.918008 8245 tablet_bootstrap.cc:654] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:07.918061 7861 tablet_bootstrap.cc:492] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:07.918136 7861 ts_tablet_manager.cc:1367] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:07.918694 7861 raft_consensus.cc:348] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.918797 7861 raft_consensus.cc:374] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.918830 7861 raft_consensus.cc:729] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.918941 7861 consensus_queue.cc:260] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.919272 7861 ts_tablet_manager.cc:1398] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:07.920141 8245 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:07.920266 8245 ts_tablet_manager.cc:1367] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:07.920856 8245 raft_consensus.cc:348] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.920964 8245 raft_consensus.cc:374] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:07.921000 8245 raft_consensus.cc:729] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:07.921115 8245 consensus_queue.cc:260] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.921399 8245 ts_tablet_manager.cc:1398] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:07.927508 8141 raft_consensus.cc:482] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.927594 8141 raft_consensus.cc:504] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.927902 8141 leader_election.cc:288] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:07.928144 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d855e5edf05471983dfdbd40f37560f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:07.928259 6691 raft_consensus.cc:2454] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.928457 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d855e5edf05471983dfdbd40f37560f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.928531 6496 leader_election.cc:302] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:07.928575 6296 raft_consensus.cc:2454] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:07.928684 8141 raft_consensus.cc:2781] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.928735 8141 raft_consensus.cc:482] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.928802 8141 raft_consensus.cc:3037] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.929517 8141 raft_consensus.cc:504] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.929821 8141 leader_election.cc:288] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:07.930902 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d855e5edf05471983dfdbd40f37560f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:07.930991 6691 raft_consensus.cc:3037] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.931649 6691 raft_consensus.cc:2454] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.931844 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d855e5edf05471983dfdbd40f37560f" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.931926 6496 leader_election.cc:302] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:07.931950 6296 raft_consensus.cc:3037] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.932096 8141 raft_consensus.cc:2781] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.932200 8141 raft_consensus.cc:686] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:07.932322 8141 consensus_queue.cc:237] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:07.932803 6296 raft_consensus.cc:2454] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:07.934019 6150 catalog_manager.cc:5484] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:07.974058 8390 raft_consensus.cc:482] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:07.974159 8390 raft_consensus.cc:504] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.974514 8390 leader_election.cc:288] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:07.974728 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cc5703d29445f691675b12fc1ce191" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:07.974776 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cc5703d29445f691675b12fc1ce191" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:07.974849 6296 raft_consensus.cc:2454] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.974884 6691 raft_consensus.cc:2454] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:07.975088 6363 leader_election.cc:302] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.975232 8390 raft_consensus.cc:2781] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:07.975286 8390 raft_consensus.cc:482] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:07.975317 8390 raft_consensus.cc:3037] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.976511 8390 raft_consensus.cc:504] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.976848 8390 leader_election.cc:288] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:07.977046 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cc5703d29445f691675b12fc1ce191" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:07.977133 6296 raft_consensus.cc:3037] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.977222 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77cc5703d29445f691675b12fc1ce191" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:07.977324 6691 raft_consensus.cc:3037] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:07.977831 6296 raft_consensus.cc:2454] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.978075 6691 raft_consensus.cc:2454] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:07.978303 6363 leader_election.cc:302] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:07.978554 8390 raft_consensus.cc:2781] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:07.978641 8390 raft_consensus.cc:686] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:07.978765 8390 consensus_queue.cc:237] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:07.980553 6150 catalog_manager.cc:5484] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:08.099633 8077 raft_consensus.cc:482] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.099752 8077 raft_consensus.cc:504] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:08.100108 8077 leader_election.cc:288] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:08.100507 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e0f5369e8349eba7188b39ae6bac30" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:08.100533 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e0f5369e8349eba7188b39ae6bac30" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:08.100643 6429 raft_consensus.cc:2454] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:08.100647 6296 raft_consensus.cc:2454] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:08.100936 6628 leader_election.cc:302] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.101161 8077 raft_consensus.cc:2781] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.101225 8077 raft_consensus.cc:482] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.101270 8077 raft_consensus.cc:3037] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.102013 8077 raft_consensus.cc:504] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:08.102300 8077 leader_election.cc:288] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:08.102540 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e0f5369e8349eba7188b39ae6bac30" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:08.102591 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e0f5369e8349eba7188b39ae6bac30" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:08.102629 6429 raft_consensus.cc:3037] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.102667 6296 raft_consensus.cc:3037] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.103272 6429 raft_consensus.cc:2454] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:08.103328 6296 raft_consensus.cc:2454] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:08.103523 6628 leader_election.cc:302] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.103672 8077 raft_consensus.cc:2781] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.103770 8077 raft_consensus.cc:686] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:08.103904 8077 consensus_queue.cc:237] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:08.105597 6150 catalog_manager.cc:5484] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:08.243269 8404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.243458 8404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.246373 8404 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 07:59:08.333822 6296 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.334123 6691 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.334510 8015 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.334650 8136 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.336561 6298 raft_consensus.cc:1261] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.338186 8391 consensus_queue.cc:1035] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.339836 6690 raft_consensus.cc:1261] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.340297 8390 consensus_queue.cc:1035] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.378387 6296 raft_consensus.cc:1261] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.378453 6429 raft_consensus.cc:1261] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.378820 8077 consensus_queue.cc:1035] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.379009 8296 consensus_queue.cc:1035] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.077 ms
time per row: 0.0645383 ms
I20240503 07:59:08.412143 6149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:08.417136 6279 tablet_service.cc:1467] Processing CreateTablet for tablet f5604af852814da58385f072eb9600aa (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.417102 6541 tablet_service.cc:1467] Processing CreateTablet for tablet f5604af852814da58385f072eb9600aa (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.417402 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5604af852814da58385f072eb9600aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.417416 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5604af852814da58385f072eb9600aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.417583 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 9772b487ca864ad98e9d79d8435c34c6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.417786 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9772b487ca864ad98e9d79d8435c34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.419250 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 9772b487ca864ad98e9d79d8435c34c6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.419415 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9772b487ca864ad98e9d79d8435c34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.419549 7861 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:08.419979 6410 tablet_service.cc:1467] Processing CreateTablet for tablet f5604af852814da58385f072eb9600aa (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.420226 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5604af852814da58385f072eb9600aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.420492 7861 tablet_bootstrap.cc:654] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.421043 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 9772b487ca864ad98e9d79d8435c34c6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.421247 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9772b487ca864ad98e9d79d8435c34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.422082 8447 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:08.422611 8449 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:08.422994 8447 tablet_bootstrap.cc:654] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.423316 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 8c01e3ce57874e24ae75a41243fdeaaf (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.423530 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c01e3ce57874e24ae75a41243fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.423530 8449 tablet_bootstrap.cc:654] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.423452 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 8c01e3ce57874e24ae75a41243fdeaaf (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.423681 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c01e3ce57874e24ae75a41243fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.424460 8449 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:08.424530 8449 ts_tablet_manager.cc:1367] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:08.424995 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 8c01e3ce57874e24ae75a41243fdeaaf (DEFAULT_TABLE table=rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:08.425029 8449 raft_consensus.cc:348] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.425132 8449 raft_consensus.cc:374] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.425160 8449 raft_consensus.cc:729] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.425221 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c01e3ce57874e24ae75a41243fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:08.425288 8449 consensus_queue.cc:260] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.425549 8449 ts_tablet_manager.cc:1398] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.425604 8447 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:08.425679 8447 ts_tablet_manager.cc:1367] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.426093 8447 raft_consensus.cc:348] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.426189 8447 raft_consensus.cc:374] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.426232 8447 raft_consensus.cc:729] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.426357 8447 consensus_queue.cc:260] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.426615 8447 ts_tablet_manager.cc:1398] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.426764 8447 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:08.426769 8449 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:08.428014 8447 tablet_bootstrap.cc:654] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.428789 8449 tablet_bootstrap.cc:654] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.428819 8450 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:08.429282 8447 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:08.429384 8447 ts_tablet_manager.cc:1367] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:08.429802 8447 raft_consensus.cc:348] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.429910 8447 raft_consensus.cc:374] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.429917 7861 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:08.429952 8447 raft_consensus.cc:729] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.429989 7861 ts_tablet_manager.cc:1367] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 07:59:08.430058 8447 consensus_queue.cc:260] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.430215 8450 tablet_bootstrap.cc:654] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.430379 8447 ts_tablet_manager.cc:1398] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:08.430527 7861 raft_consensus.cc:348] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.430634 7861 raft_consensus.cc:374] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.430665 7861 raft_consensus.cc:729] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.430761 7861 consensus_queue.cc:260] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.431011 7861 ts_tablet_manager.cc:1398] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.431149 7861 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:08.431222 8447 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:08.431933 7861 tablet_bootstrap.cc:654] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.432101 8450 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:08.432166 8450 ts_tablet_manager.cc:1367] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.432184 8447 tablet_bootstrap.cc:654] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.432531 8450 raft_consensus.cc:348] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.432636 8450 raft_consensus.cc:374] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.432668 8450 raft_consensus.cc:729] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.432790 8450 consensus_queue.cc:260] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.433089 8450 ts_tablet_manager.cc:1398] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.433248 8450 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:08.433996 8447 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:08.434013 8450 tablet_bootstrap.cc:654] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:08.434067 8447 ts_tablet_manager.cc:1367] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:08.434406 8447 raft_consensus.cc:348] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.434491 8447 raft_consensus.cc:374] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.434528 8447 raft_consensus.cc:729] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.434638 8447 consensus_queue.cc:260] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.434868 8447 ts_tablet_manager.cc:1398] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.435415 8449 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:08.435637 8449 ts_tablet_manager.cc:1367] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:08.436065 8449 raft_consensus.cc:348] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.436192 8449 raft_consensus.cc:374] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.436244 8449 raft_consensus.cc:729] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.436381 8449 consensus_queue.cc:260] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.436688 8449 ts_tablet_manager.cc:1398] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.438186 8450 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:08.438266 8450 ts_tablet_manager.cc:1367] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:08.438659 8450 raft_consensus.cc:348] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.438854 8450 raft_consensus.cc:374] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.438890 8450 raft_consensus.cc:729] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.438997 8450 consensus_queue.cc:260] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.439218 8450 ts_tablet_manager.cc:1398] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.439849 7861 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:08.439939 7861 ts_tablet_manager.cc:1367] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:08.440337 7861 raft_consensus.cc:348] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.440451 7861 raft_consensus.cc:374] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:08.440483 7861 raft_consensus.cc:729] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:08.440585 7861 consensus_queue.cc:260] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.440867 7861 ts_tablet_manager.cc:1398] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:08.465425 8390 raft_consensus.cc:482] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.465500 8390 raft_consensus.cc:504] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.465795 8390 leader_election.cc:288] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:08.466045 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c01e3ce57874e24ae75a41243fdeaaf" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:08.466063 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c01e3ce57874e24ae75a41243fdeaaf" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:08.466136 6560 raft_consensus.cc:2454] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:08.466161 6691 raft_consensus.cc:2454] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:08.466383 6363 leader_election.cc:302] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.466573 8390 raft_consensus.cc:2781] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.466614 8390 raft_consensus.cc:482] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.466648 8390 raft_consensus.cc:3037] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.467279 8390 raft_consensus.cc:504] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.467554 8390 leader_election.cc:288] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:08.467814 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c01e3ce57874e24ae75a41243fdeaaf" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:08.467901 6560 raft_consensus.cc:3037] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.468546 6560 raft_consensus.cc:2454] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:08.468760 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c01e3ce57874e24ae75a41243fdeaaf" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:08.468842 6363 leader_election.cc:302] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.468901 6691 raft_consensus.cc:3037] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.468999 8390 raft_consensus.cc:2781] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.469089 8390 raft_consensus.cc:686] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:08.469202 8390 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.469656 6691 raft_consensus.cc:2454] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:08.470831 6149 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.520864 8296 raft_consensus.cc:482] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.520961 8296 raft_consensus.cc:504] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.521239 8296 leader_election.cc:288] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:08.521500 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9772b487ca864ad98e9d79d8435c34c6" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:08.521546 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9772b487ca864ad98e9d79d8435c34c6" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:08.521642 6560 raft_consensus.cc:2454] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:08.521638 6298 raft_consensus.cc:2454] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:08.521857 6625 leader_election.cc:302] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:08.522019 8077 raft_consensus.cc:2781] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.522084 8077 raft_consensus.cc:482] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.522121 8077 raft_consensus.cc:3037] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.522904 8077 raft_consensus.cc:504] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.523283 8077 leader_election.cc:288] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:08.523530 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9772b487ca864ad98e9d79d8435c34c6" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:08.523654 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9772b487ca864ad98e9d79d8435c34c6" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:08.523736 6560 raft_consensus.cc:3037] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.523635 6298 raft_consensus.cc:3037] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.524574 6560 raft_consensus.cc:2454] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:08.524588 6298 raft_consensus.cc:2454] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:08.524922 6625 leader_election.cc:302] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:08.525130 8077 raft_consensus.cc:2781] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.525224 8077 raft_consensus.cc:686] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:08.525359 8077 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:08.527259 6149 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:08.552238 8390 raft_consensus.cc:482] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:08.552328 8390 raft_consensus.cc:504] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.552642 8390 leader_election.cc:288] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:08.552965 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5604af852814da58385f072eb9600aa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:08.552989 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5604af852814da58385f072eb9600aa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:08.553107 6298 raft_consensus.cc:2454] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:08.553064 6560 raft_consensus.cc:2454] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:08.553356 6363 leader_election.cc:302] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.553617 8390 raft_consensus.cc:2781] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:08.553671 8390 raft_consensus.cc:482] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:08.553696 8390 raft_consensus.cc:3037] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.554438 8390 raft_consensus.cc:504] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.554708 8390 leader_election.cc:288] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:08.554996 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5604af852814da58385f072eb9600aa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:08.555068 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5604af852814da58385f072eb9600aa" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:08.555130 6560 raft_consensus.cc:3037] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.555163 6298 raft_consensus.cc:3037] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:08.556049 6298 raft_consensus.cc:2454] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:08.556049 6560 raft_consensus.cc:2454] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:08.556351 6363 leader_election.cc:302] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:08.556535 8312 raft_consensus.cc:2781] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:08.556632 8312 raft_consensus.cc:686] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:08.556761 8312 consensus_queue.cc:237] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:08.558699 6149 catalog_manager.cc:5484] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:08.687521 8462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.687742 8462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.690726 8462 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 07:59:08.767941 6690 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.767944 6561 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.768364 8390 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.768818 6298 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.769348 8391 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.769484 8390 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.776700 6560 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.777475 8391 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.795506 6296 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.796073 6561 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:08.796541 8077 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:08.796912 8296 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.203 ms
time per row: 0.0556017 ms
I20240503 07:59:08.849692 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:39949
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:39595
--webserver_port=35381
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:08.959712 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:08.959949 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:08.960034 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:08.963083 8499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:08.963162 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:08.963268 8499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:08.966989 8499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:39949
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=35381
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.8499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:08.967830 8499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:08.968915 8499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:08.977651 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:08.978462 8499 server_base.cc:1034] running on GCE node
W20240503 07:59:08.978950 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:08.979620 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:08.979980 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:08.980502 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:08.981673 8499 hybrid_clock.cc:648] HybridClock initialized: now 1714723148981650 us; error 33 us; skew 500 ppm
I20240503 07:59:08.983428 8499 webserver.cc:460] Webserver started at http://127.0.99.133:35381/ using document root <none> and password file <none>
I20240503 07:59:08.983958 8499 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:08.984020 8499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:08.987450 8499 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:08.991293 8513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:08.992362 8499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:08.992486 8499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c8d71787690245e681c85a087086e7ed"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:59:08.992924 8499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:09.016301 8499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:09.016844 8499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:09.016978 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:09.017452 8499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:09.018373 8499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:09.018456 8499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.018508 8499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:09.018540 8499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.027110 8499 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:39949
I20240503 07:59:09.027146 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:39949 every 8 connection(s)
I20240503 07:59:09.028105 8499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:09.035794 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 8499
I20240503 07:59:09.036163 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:40925
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:39595
--webserver_port=44279
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:09.036504 8627 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:59:09.036682 8627 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:09.037201 8627 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:59:09.038268 6149 ts_manager.cc:194] Re-registered known tserver with Master: c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:09.038796 6149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:53619
W20240503 07:59:09.140574 8630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.140733 8630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:09.140800 8630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.144114 8630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:09.144172 8630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:09.144253 8630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:09.148284 8630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:40925
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=44279
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.8630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:09.149129 8630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:09.150259 8630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:09.158766 8635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.159265 8638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.160056 8630 server_base.cc:1034] running on GCE node
W20240503 07:59:09.160537 8636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.160946 8630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:09.161502 8630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:09.162671 8630 hybrid_clock.cc:648] HybridClock initialized: now 1714723149162656 us; error 29 us; skew 500 ppm
I20240503 07:59:09.164552 8630 webserver.cc:460] Webserver started at http://127.0.99.134:44279/ using document root <none> and password file <none>
I20240503 07:59:09.165086 8630 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:09.165158 8630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:09.168471 8630 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:09.170614 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.171581 8630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:09.171684 8630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:59:09.172008 8630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:09.220522 8630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:09.221119 8630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:09.221266 8630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:09.221781 8630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:09.222622 8630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:09.222667 8630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.222718 8630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:09.222741 8630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.230823 8630 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:40925
I20240503 07:59:09.230857 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:40925 every 8 connection(s)
I20240503 07:59:09.231732 8630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:09.239640 8758 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:59:09.240120 8758 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:09.240701 8758 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:59:09.241535 6149 ts_manager.cc:194] Re-registered known tserver with Master: 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:09.241910 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 8630
I20240503 07:59:09.242048 6149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:42669
I20240503 07:59:09.242290 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:39417
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:39595
--webserver_port=33603
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:09.350046 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.350277 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:09.350348 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.353420 8761 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:09.353477 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:09.353550 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:09.357144 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:39417
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=33603
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.8761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:09.357878 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:09.358835 8761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:09.367064 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.367754 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.368374 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.369118 8761 server_base.cc:1034] running on GCE node
I20240503 07:59:09.369488 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:09.370015 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:09.371170 8761 hybrid_clock.cc:648] HybridClock initialized: now 1714723149371156 us; error 31 us; skew 500 ppm
I20240503 07:59:09.372880 8761 webserver.cc:460] Webserver started at http://127.0.99.135:33603/ using document root <none> and password file <none>
I20240503 07:59:09.373391 8761 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:09.373456 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:09.376639 8761 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 07:59:09.378278 8775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.379153 8761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:09.379252 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "616fdda6ba714cb7be01a564de2b9db2"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:59:09.379613 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:09.409032 8761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:09.409574 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:09.409708 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:09.410189 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:09.410969 8761 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:09.411031 8761 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.411087 8761 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:09.411115 8761 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.419360 8761 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:39417
I20240503 07:59:09.419410 8889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:39417 every 8 connection(s)
I20240503 07:59:09.420287 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:09.427188 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 8761
I20240503 07:59:09.427641 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41089
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:39595
--webserver_port=41261
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:09.428239 8890 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:59:09.428427 8890 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:09.428961 8890 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:59:09.429805 6149 ts_manager.cc:194] Re-registered known tserver with Master: 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:09.430290 6149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:51647
W20240503 07:59:09.530627 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.530810 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:09.530864 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.533730 8894 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:09.533787 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:09.533869 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:09.537328 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41089
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=41261
--tserver_master_addrs=127.0.99.190:39595
--never_fsync=true
--heap_profile_path=/tmp/kudu.8894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:09.538038 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:09.539038 8894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:09.548635 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:09.549190 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.549841 8894 server_base.cc:1034] running on GCE node
W20240503 07:59:09.550339 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:09.550791 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:09.551407 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:09.552574 8894 hybrid_clock.cc:648] HybridClock initialized: now 1714723149552556 us; error 33 us; skew 500 ppm
I20240503 07:59:09.554495 8894 webserver.cc:460] Webserver started at http://127.0.99.136:41261/ using document root <none> and password file <none>
I20240503 07:59:09.555115 8894 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:09.555183 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:09.558444 8894 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:09.560170 8909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.561278 8894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:09.561390 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "3d08a06d53b14381bee6223c4e1a6027"
format_stamp: "Formatted at 2024-05-03 07:58:54 on dist-test-slave-7tt0"
I20240503 07:59:09.561790 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:09.581705 8894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:09.582249 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:09.582397 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:09.582914 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:09.583746 8894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:09.583797 8894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.583863 8894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:09.583894 8894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:09.593012 8894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41089
I20240503 07:59:09.593055 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41089 every 8 connection(s)
I20240503 07:59:09.594002 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:09.601750 9023 heartbeater.cc:347] Connected to a master server at 127.0.99.190:39595
I20240503 07:59:09.601943 9023 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:09.602418 9023 heartbeater.cc:511] Master 127.0.99.190:39595 requested a full tablet report, sending...
I20240503 07:59:09.602699 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 8894
I20240503 07:59:09.604894 6149 ts_manager.cc:194] Re-registered known tserver with Master: 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:09.605594 6149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:44667
I20240503 07:59:09.605907 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:09.616487 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]), partition=
I20240503 07:59:09.616722 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602fe180c2a44a799a8acbce3f99c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.618906 9038 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:09.619971 9038 tablet_bootstrap.cc:654] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.621364 9038 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:09.621446 9038 ts_tablet_manager.cc:1367] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:09.622154 9038 raft_consensus.cc:348] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.622287 9038 raft_consensus.cc:374] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.622325 9038 raft_consensus.cc:729] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.622444 9038 consensus_queue.cc:260] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.622700 9038 ts_tablet_manager.cc:1398] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:09.631556 8561 tablet_service.cc:1467] Processing CreateTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]), partition=
I20240503 07:59:09.632126 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602fe180c2a44a799a8acbce3f99c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.633344 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]), partition=
I20240503 07:59:09.633939 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9602fe180c2a44a799a8acbce3f99c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:09.638480 9041 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:09.640429 9041 tablet_bootstrap.cc:654] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.641356 9041 log.cc:824] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:09.642206 9042 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:09.643711 9042 tablet_bootstrap.cc:654] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:09.644315 9042 log.cc:824] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:09.645788 9041 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:09.645951 9042 tablet_bootstrap.cc:492] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:09.646122 9042 ts_tablet_manager.cc:1367] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:09.646004 9041 ts_tablet_manager.cc:1367] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 07:59:09.649029 9041 raft_consensus.cc:348] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.649204 9041 raft_consensus.cc:374] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.649302 9041 raft_consensus.cc:729] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.649487 9042 raft_consensus.cc:348] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.649667 9042 raft_consensus.cc:374] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:09.649717 9042 raft_consensus.cc:729] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:09.649688 9041 consensus_queue.cc:260] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.650121 9042 consensus_queue.cc:260] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.650259 8627 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:59:09.650485 9041 ts_tablet_manager.cc:1398] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:09.650832 9042 ts_tablet_manager.cc:1398] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:09.651170 8890 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
W20240503 07:59:09.671028 8891 tablet.cc:2369] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:09.705942 9045 raft_consensus.cc:482] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:09.706175 9045 raft_consensus.cc:504] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.707288 9045 leader_election.cc:288] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:09.711627 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602fe180c2a44a799a8acbce3f99c95" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:09.711892 8844 raft_consensus.cc:2454] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:09.712306 8516 leader_election.cc:302] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:09.712550 9045 raft_consensus.cc:2781] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:09.712604 9045 raft_consensus.cc:482] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:09.712666 9045 raft_consensus.cc:3037] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.713119 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602fe180c2a44a799a8acbce3f99c95" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:09.713248 6430 raft_consensus.cc:2454] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:09.713721 9045 raft_consensus.cc:504] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.714040 9045 leader_election.cc:288] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:09.714296 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602fe180c2a44a799a8acbce3f99c95" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:09.714352 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9602fe180c2a44a799a8acbce3f99c95" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:09.714407 8844 raft_consensus.cc:3037] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.714437 6430 raft_consensus.cc:3037] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:09.715292 8844 raft_consensus.cc:2454] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:09.715292 6430 raft_consensus.cc:2454] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:09.715569 8516 leader_election.cc:302] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:09.715740 9045 raft_consensus.cc:2781] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:09.715914 9045 raft_consensus.cc:686] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:09.716233 9045 consensus_queue.cc:237] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:09.719046 6150 catalog_manager.cc:5484] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:09.779304 8628 tablet.cc:2369] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:09.826609 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:09.833613 6430 raft_consensus.cc:1261] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.834019 8844 raft_consensus.cc:1261] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:09.834355 9050 consensus_queue.cc:1035] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.834748 9050 consensus_queue.cc:1035] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:09.838339 6149 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]). Expected version 1 got 0
I20240503 07:59:09.839246 6149 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]). Expected version 1 got 0
I20240503 07:59:09.840005 9092 mvcc.cc:205] Tried to move back new op lower bound from 7023506021709844480 to 7023506021240074240. Current Snapshot: MvccSnapshot[applied={T|T < 7023506021709844480}]
I20240503 07:59:09.840306 9112 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:09.841256 6149 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]). Expected version 1 got 0
I20240503 07:59:09.844825 6155 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]). Expected version 1 got 0
I20240503 07:59:09.850373 6149 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced]). Expected version 1 got 0
I20240503 07:59:09.851025 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced] alter complete (version 1)
I20240503 07:59:09.851712 8824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:09.852846 9112 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:09.853346 9113 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:09.854029 9113 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:09.855823 9110 mvcc.cc:205] Tried to move back new op lower bound from 7023506021709844480 to 7023506021240074240. Current Snapshot: MvccSnapshot[applied={T|T < 7023506021709844480}]
I20240503 07:59:09.868084 8957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:09.872550 6671 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:09.873217 9114 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:09.873342 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:09.874032 9114 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:09.875762 6408 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:09.879622 8692 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:09.880175 8561 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:09.885154 6278 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:09.953647 6152 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:09.959115 9112 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:09.959390 9114 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:09.959442 9112 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:09.959646 9114 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:09.960732 9113 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:09.961017 9113 tablet.cc:1620] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:09.963979 6152 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7b84e8be3b224691a4c6a2bc6ea56ced] alter complete (version 2)
I20240503 07:59:10.043437 8957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.043529 8824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.044121 8692 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.056422 6671 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:10.063817 6278 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:10.066174 6408 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:10.067291 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:10.072958 9112 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.074635 9114 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.078045 9113 tablet.cc:1722] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.080233 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=7b84e8be3b224691a4c6a2bc6ea56ced] alter complete (version 3)
I20240503 07:59:10.084035 8561 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:10.089226 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:10.103996 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 07:59:10.104040 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:10.104153 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 07:59:10.108341 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet d5dd030945264526a0570de935618ff5
I20240503 07:59:10.108670 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet f5b2f846e84c4dbc9fab564845def15d
I20240503 07:59:10.109218 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd030945264526a0570de935618ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:10.109942 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 71b9249ed9a94cb2a91bb79ba5d077b2
I20240503 07:59:10.110049 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd030945264526a0570de935618ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:10.110118 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd030945264526a0570de935618ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.110611 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:10.110615 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b2f846e84c4dbc9fab564845def15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:10.111145 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b2f846e84c4dbc9fab564845def15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:10.111233 9175 tablet_replica.cc:330] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:10.111313 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.111496 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b9249ed9a94cb2a91bb79ba5d077b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:10.111541 9175 raft_consensus.cc:2229] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.111646 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b2f846e84c4dbc9fab564845def15d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.111897 9175 raft_consensus.cc:2258] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.112936 9175 ts_tablet_manager.cc:1875] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.112970 9176 tablet_replica.cc:330] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:10.113304 9176 raft_consensus.cc:2229] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.113526 9176 raft_consensus.cc:2258] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.114432 9176 ts_tablet_manager.cc:1875] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.114650 9178 tablet_replica.cc:330] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:10.114868 8824 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4ca6638764fb49f3afcab7688e2b3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.115221 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4ca6638764fb49f3afcab7688e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.115485 9178 raft_consensus.cc:2229] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.115659 9178 raft_consensus.cc:2258] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.116163 8561 tablet_service.cc:1467] Processing CreateTablet for tablet c8cf3d6a3ace46769854ed61f340fb25 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.116365 9178 ts_tablet_manager.cc:1875] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.116478 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cf3d6a3ace46769854ed61f340fb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.118463 6671 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4ca6638764fb49f3afcab7688e2b3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.118528 9178 ts_tablet_manager.cc:1888] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.118603 9178 log.cc:1196] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f5b2f846e84c4dbc9fab564845def15d
I20240503 07:59:10.118609 9041 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:10.118693 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4ca6638764fb49f3afcab7688e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.118878 9178 ts_tablet_manager.cc:1909] T f5b2f846e84c4dbc9fab564845def15d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:10.119347 9041 tablet_bootstrap.cc:654] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.116164 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 5060df68b95e48a8a849e77ba926e608 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.120162 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5060df68b95e48a8a849e77ba926e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.121838 8823 tablet_service.cc:1467] Processing CreateTablet for tablet c8cf3d6a3ace46769854ed61f340fb25 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.122076 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cf3d6a3ace46769854ed61f340fb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.121865 8692 tablet_service.cc:1467] Processing CreateTablet for tablet c8cf3d6a3ace46769854ed61f340fb25 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.122473 9180 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:10.122748 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cf3d6a3ace46769854ed61f340fb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.123142 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet f5b2f846e84c4dbc9fab564845def15d (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.123441 9180 tablet_bootstrap.cc:654] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.123973 9177 tablet_replica.cc:330] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.124060 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 5060df68b95e48a8a849e77ba926e608 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.124217 9177 raft_consensus.cc:2229] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.124270 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5060df68b95e48a8a849e77ba926e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.124401 9177 raft_consensus.cc:2258] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.125581 9177 ts_tablet_manager.cc:1875] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.126536 9042 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:10.127339 9042 tablet_bootstrap.cc:654] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.127671 9176 ts_tablet_manager.cc:1888] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.127743 9176 log.cc:1196] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5dd030945264526a0570de935618ff5
I20240503 07:59:10.127460 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 5060df68b95e48a8a849e77ba926e608 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.127957 9177 ts_tablet_manager.cc:1888] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.128013 9176 ts_tablet_manager.cc:1909] T d5dd030945264526a0570de935618ff5 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:10.128170 9177 log.cc:1196] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d5dd030945264526a0570de935618ff5
I20240503 07:59:10.128302 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5060df68b95e48a8a849e77ba926e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.128495 9177 ts_tablet_manager.cc:1909] T d5dd030945264526a0570de935618ff5 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.128932 9176 tablet_replica.cc:330] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:10.128957 8957 tablet_service.cc:1467] Processing CreateTablet for tablet f4e4ca6638764fb49f3afcab7688e2b3 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:10.128983 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet d5dd030945264526a0570de935618ff5 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.129076 9176 raft_consensus.cc:2229] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.129176 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4ca6638764fb49f3afcab7688e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.129192 9176 raft_consensus.cc:2258] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.129434 9175 ts_tablet_manager.cc:1888] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.129504 9175 log.cc:1196] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5dd030945264526a0570de935618ff5
I20240503 07:59:10.129650 9176 ts_tablet_manager.cc:1875] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.129751 9175 ts_tablet_manager.cc:1909] T d5dd030945264526a0570de935618ff5 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:10.130667 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet d5dd030945264526a0570de935618ff5 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.130761 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet d5dd030945264526a0570de935618ff5 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.130863 9177 tablet_replica.cc:330] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.130861 9178 tablet_replica.cc:330] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:10.130970 9177 raft_consensus.cc:2229] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.130990 9178 raft_consensus.cc:2229] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.131047 9175 tablet_replica.cc:330] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:10.131098 9178 raft_consensus.cc:2258] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.131153 9175 raft_consensus.cc:2229] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.131181 9177 raft_consensus.cc:2258] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.131256 9175 raft_consensus.cc:2258] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.131572 9178 ts_tablet_manager.cc:1875] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.131634 9177 ts_tablet_manager.cc:1875] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.131650 9175 ts_tablet_manager.cc:1875] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.132747 9042 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:10.132851 9042 ts_tablet_manager.cc:1367] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:10.133302 9042 raft_consensus.cc:348] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.133416 9042 raft_consensus.cc:374] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.133430 9178 ts_tablet_manager.cc:1888] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.133455 9042 raft_consensus.cc:729] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.133489 9178 log.cc:1196] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/71b9249ed9a94cb2a91bb79ba5d077b2
I20240503 07:59:10.133570 9042 consensus_queue.cc:260] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.133822 9042 ts_tablet_manager.cc:1398] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:10.134007 9042 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:10.134281 9176 ts_tablet_manager.cc:1888] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.134342 9176 log.cc:1196] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5b2f846e84c4dbc9fab564845def15d
I20240503 07:59:10.134570 9176 ts_tablet_manager.cc:1909] T f5b2f846e84c4dbc9fab564845def15d P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:10.134792 9042 tablet_bootstrap.cc:654] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.134896 9178 ts_tablet_manager.cc:1909] T 71b9249ed9a94cb2a91bb79ba5d077b2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:10.135418 9177 ts_tablet_manager.cc:1888] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.135473 9177 log.cc:1196] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/71b9249ed9a94cb2a91bb79ba5d077b2
I20240503 07:59:10.135458 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.135682 9177 ts_tablet_manager.cc:1909] T 71b9249ed9a94cb2a91bb79ba5d077b2 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.135854 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet f5b2f846e84c4dbc9fab564845def15d (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.135881 9041 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:10.135969 9041 ts_tablet_manager.cc:1367] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 07:59:10.136394 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.136457 9041 raft_consensus.cc:348] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.136543 9041 raft_consensus.cc:374] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.136562 9042 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:10.136574 9041 raft_consensus.cc:729] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.136637 9042 ts_tablet_manager.cc:1367] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:10.136683 9041 consensus_queue.cc:260] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.136916 9177 tablet_replica.cc:330] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.136943 9041 ts_tablet_manager.cc:1398] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.137032 9177 raft_consensus.cc:2229] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.137084 9041 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:10.137125 9042 raft_consensus.cc:348] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.137212 9042 raft_consensus.cc:374] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.137244 9042 raft_consensus.cc:729] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.137343 9042 consensus_queue.cc:260] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.137387 9177 raft_consensus.cc:2258] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.137787 9177 ts_tablet_manager.cc:1875] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.137861 9041 tablet_bootstrap.cc:654] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.138089 9042 ts_tablet_manager.cc:1398] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.138234 9042 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:10.139029 9042 tablet_bootstrap.cc:654] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.139415 9177 ts_tablet_manager.cc:1888] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.139523 9177 log.cc:1196] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f5b2f846e84c4dbc9fab564845def15d
I20240503 07:59:10.139752 9177 ts_tablet_manager.cc:1909] T f5b2f846e84c4dbc9fab564845def15d P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.140450 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet f5b2f846e84c4dbc9fab564845def15d (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.140535 9175 ts_tablet_manager.cc:1888] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.140594 9175 log.cc:1196] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/71b9249ed9a94cb2a91bb79ba5d077b2
I20240503 07:59:10.140866 9175 ts_tablet_manager.cc:1909] T 71b9249ed9a94cb2a91bb79ba5d077b2 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:10.141081 9042 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:10.141197 9042 ts_tablet_manager.cc:1367] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:10.141301 9041 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:10.141381 9041 ts_tablet_manager.cc:1367] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:10.141520 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 71b9249ed9a94cb2a91bb79ba5d077b2 (table rebalance_test_table_0 [id=3b991e2f55434f818f4fc17177c9137e]) successfully deleted
I20240503 07:59:10.141649 9042 raft_consensus.cc:348] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.141728 9042 raft_consensus.cc:374] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.141754 9042 raft_consensus.cc:729] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.141861 9042 consensus_queue.cc:260] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.141899 9041 raft_consensus.cc:348] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.141983 9041 raft_consensus.cc:374] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.142012 9041 raft_consensus.cc:729] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.142097 9042 ts_tablet_manager.cc:1398] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:10.142108 9041 consensus_queue.cc:260] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.142352 9041 ts_tablet_manager.cc:1398] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.143106 9180 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:10.143206 9180 ts_tablet_manager.cc:1367] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:10.143601 9180 raft_consensus.cc:348] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.143697 9180 raft_consensus.cc:374] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.143729 9180 raft_consensus.cc:729] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.143860 9180 consensus_queue.cc:260] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.144129 9180 ts_tablet_manager.cc:1398] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.150393 9186 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:10.152423 9186 tablet_bootstrap.cc:654] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.153133 9186 log.cc:824] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.155360 9186 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:10.155519 9186 ts_tablet_manager.cc:1367] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:10.158494 9186 raft_consensus.cc:348] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.158686 9186 raft_consensus.cc:374] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.158728 9186 raft_consensus.cc:729] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.159121 9186 consensus_queue.cc:260] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.159695 8758 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:59:10.159885 9186 ts_tablet_manager.cc:1398] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 07:59:10.160255 9188 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:10.162057 9188 tablet_bootstrap.cc:654] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.162777 9188 log.cc:824] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:10.164219 9188 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:10.164362 9188 ts_tablet_manager.cc:1367] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:10.167063 9188 raft_consensus.cc:348] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.167218 9188 raft_consensus.cc:374] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.167256 9188 raft_consensus.cc:729] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.167634 9188 consensus_queue.cc:260] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.168334 9046 raft_consensus.cc:482] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.168437 9188 ts_tablet_manager.cc:1398] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:10.168431 9046 raft_consensus.cc:504] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.168745 9188 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:10.168756 9023 heartbeater.cc:503] Master 127.0.99.190:39595 was elected leader, sending a full tablet report...
I20240503 07:59:10.169631 9046 leader_election.cc:288] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:10.170010 9188 tablet_bootstrap.cc:654] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.171154 9188 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:10.171293 9188 ts_tablet_manager.cc:1367] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:10.171808 9188 raft_consensus.cc:348] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.171968 9188 raft_consensus.cc:374] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.172048 9188 raft_consensus.cc:729] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.172188 9188 consensus_queue.cc:260] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.172891 9188 ts_tablet_manager.cc:1398] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:10.176704 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5060df68b95e48a8a849e77ba926e608" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:10.177004 8977 raft_consensus.cc:2454] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:10.177388 8778 leader_election.cc:302] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:10.177595 9046 raft_consensus.cc:2781] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:10.177649 9046 raft_consensus.cc:482] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:10.177682 9046 raft_consensus.cc:3037] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.177815 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5060df68b95e48a8a849e77ba926e608" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:10.177955 8575 raft_consensus.cc:2454] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:10.178393 9046 raft_consensus.cc:504] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.178653 9046 leader_election.cc:288] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:10.178944 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5060df68b95e48a8a849e77ba926e608" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:10.178969 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5060df68b95e48a8a849e77ba926e608" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:10.179057 8977 raft_consensus.cc:3037] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.179064 8575 raft_consensus.cc:3037] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.179834 8977 raft_consensus.cc:2454] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:10.180019 8575 raft_consensus.cc:2454] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:10.180075 8778 leader_election.cc:302] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:10.180212 9046 raft_consensus.cc:2781] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.180377 9046 raft_consensus.cc:686] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:10.180603 9046 consensus_queue.cc:237] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:10.182271 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:10.182368 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:10.182504 6157 catalog_manager.cc:5484] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:10.184954 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 9602fe180c2a44a799a8acbce3f99c95
I20240503 07:59:10.185585 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:10.185639 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:10.185863 9178 tablet_replica.cc:330] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:10.185978 9178 raft_consensus.cc:2229] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.186069 9178 raft_consensus.cc:2258] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.186215 9208 tablet_replica.cc:330] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:10.186272 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 9602fe180c2a44a799a8acbce3f99c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:10.186566 9208 raft_consensus.cc:2229] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.186692 9209 tablet_replica.cc:330] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:10.186779 9208 raft_consensus.cc:2258] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.187014 9178 ts_tablet_manager.cc:1875] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.187078 9209 raft_consensus.cc:2229] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.187409 9209 raft_consensus.cc:2258] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.187855 9208 ts_tablet_manager.cc:1875] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.188407 9209 ts_tablet_manager.cc:1875] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.188971 9178 ts_tablet_manager.cc:1888] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.189045 9178 log.cc:1196] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9602fe180c2a44a799a8acbce3f99c95
I20240503 07:59:10.189272 9178 ts_tablet_manager.cc:1909] T 9602fe180c2a44a799a8acbce3f99c95 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:10.189844 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table_new_name [id=7b84e8be3b224691a4c6a2bc6ea56ced]) successfully deleted
I20240503 07:59:10.190184 9208 ts_tablet_manager.cc:1888] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.190276 9208 log.cc:1196] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9602fe180c2a44a799a8acbce3f99c95
I20240503 07:59:10.190698 9208 ts_tablet_manager.cc:1909] T 9602fe180c2a44a799a8acbce3f99c95 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:10.191023 9209 ts_tablet_manager.cc:1888] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.191131 9209 log.cc:1196] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9602fe180c2a44a799a8acbce3f99c95
I20240503 07:59:10.191437 9209 ts_tablet_manager.cc:1909] T 9602fe180c2a44a799a8acbce3f99c95 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:10.191845 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table_new_name [id=7b84e8be3b224691a4c6a2bc6ea56ced]) successfully deleted
I20240503 07:59:10.192442 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 9602fe180c2a44a799a8acbce3f99c95 (table rebalancer_dynamic_table_new_name [id=7b84e8be3b224691a4c6a2bc6ea56ced]) successfully deleted
W20240503 07:59:10.233012 8759 tablet.cc:2369] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:10.286793 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:10.292690 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]), partition=
I20240503 07:59:10.293001 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7813cfc219a64d6f8d3678a053f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.295264 9042 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:10.296097 9042 tablet_bootstrap.cc:654] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.296815 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]), partition=
I20240503 07:59:10.297040 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7813cfc219a64d6f8d3678a053f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.298568 9186 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:10.299144 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]), partition=
I20240503 07:59:10.299255 9186 tablet_bootstrap.cc:654] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.299361 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7813cfc219a64d6f8d3678a053f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.299767 9042 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:10.299855 9042 ts_tablet_manager.cc:1367] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:10.300144 9186 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:10.300199 9186 ts_tablet_manager.cc:1367] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:10.300251 9042 raft_consensus.cc:348] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.300343 9042 raft_consensus.cc:374] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.300374 9042 raft_consensus.cc:729] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.300491 9042 consensus_queue.cc:260] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.300666 9186 raft_consensus.cc:348] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.300740 9042 ts_tablet_manager.cc:1398] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:10.300758 9186 raft_consensus.cc:374] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.300815 9186 raft_consensus.cc:729] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.300944 9186 consensus_queue.cc:260] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.301193 9186 ts_tablet_manager.cc:1398] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:10.302424 9243 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:10.303319 9243 tablet_bootstrap.cc:654] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.304219 9243 tablet_bootstrap.cc:492] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:10.304363 9243 ts_tablet_manager.cc:1367] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.304720 9243 raft_consensus.cc:348] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.304845 9243 raft_consensus.cc:374] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.305006 9243 raft_consensus.cc:729] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.305142 9243 consensus_queue.cc:260] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.305423 9243 ts_tablet_manager.cc:1398] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.343789 9046 raft_consensus.cc:482] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.343883 9046 raft_consensus.cc:504] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.344303 9046 leader_election.cc:288] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
W20240503 07:59:10.345260 9024 tablet.cc:2369] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:10.347836 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7813cfc219a64d6f8d3678a053f82b5a" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:10.347947 6296 raft_consensus.cc:2454] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:10.348220 8776 leader_election.cc:302] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:10.348410 9046 raft_consensus.cc:2781] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:10.348493 9046 raft_consensus.cc:482] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:10.348541 9046 raft_consensus.cc:3037] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.348706 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7813cfc219a64d6f8d3678a053f82b5a" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:10.348959 8712 raft_consensus.cc:2454] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:10.349465 9046 raft_consensus.cc:504] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.349777 9046 leader_election.cc:288] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:10.350006 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7813cfc219a64d6f8d3678a053f82b5a" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:10.350116 6296 raft_consensus.cc:3037] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.350229 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7813cfc219a64d6f8d3678a053f82b5a" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:10.350329 8712 raft_consensus.cc:3037] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.350934 6296 raft_consensus.cc:2454] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:10.351166 8776 leader_election.cc:302] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:10.351307 9046 raft_consensus.cc:2781] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.351372 8712 raft_consensus.cc:2454] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:10.351418 9046 raft_consensus.cc:686] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:10.351537 9046 consensus_queue.cc:237] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.353053 9189 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.353137 9189 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.353121 6156 catalog_manager.cc:5484] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.354177 9189 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:10.358013 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:10.358129 8844 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:10.358202 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:10.358338 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:10.358471 8647 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:10.358717 9189 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:10.358774 9189 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:10.358808 9189 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.359648 9189 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.359920 9189 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:10.360193 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:10.360194 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:10.360299 8575 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.360299 8844 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.361171 8844 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:10.361171 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:10.361430 8645 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:10.361578 9189 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.361742 9189 raft_consensus.cc:686] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:10.362002 9189 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:10.364712 6156 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:10.439750 8296 raft_consensus.cc:482] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:10.439847 8296 raft_consensus.cc:504] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.440246 8296 leader_election.cc:288] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:10.443748 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4ca6638764fb49f3afcab7688e2b3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:10.443807 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4ca6638764fb49f3afcab7688e2b3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:10.443876 8844 raft_consensus.cc:2454] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:10.443967 8977 raft_consensus.cc:2454] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:10.444118 6627 leader_election.cc:302] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:10.444286 8296 raft_consensus.cc:2781] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:10.444347 8296 raft_consensus.cc:482] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:10.444380 8296 raft_consensus.cc:3037] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.445278 8296 raft_consensus.cc:504] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.445626 8296 leader_election.cc:288] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:10.445936 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4ca6638764fb49f3afcab7688e2b3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:10.445950 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e4ca6638764fb49f3afcab7688e2b3" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:10.446033 8844 raft_consensus.cc:3037] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.446072 8977 raft_consensus.cc:3037] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:10.446851 8844 raft_consensus.cc:2454] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:10.446957 8977 raft_consensus.cc:2454] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:10.447103 6627 leader_election.cc:302] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:10.447265 8296 raft_consensus.cc:2781] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:10.447427 8296 raft_consensus.cc:686] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:10.447561 8296 consensus_queue.cc:237] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.449368 6156 catalog_manager.cc:5484] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.462469 8956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.466931 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:10.468072 8692 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:10.469255 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:10.471130 8824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:10.474433 8712 raft_consensus.cc:1261] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:10.475332 9046 consensus_queue.cc:1035] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.476199 6671 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 07:59:10.477483 6296 raft_consensus.cc:1261] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:10.477864 9046 consensus_queue.cc:1035] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.480568 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]). Expected version 1 got 0
I20240503 07:59:10.482757 9114 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:10.483045 9114 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:10.484695 6278 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:10.487129 6156 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]). Expected version 1 got 0
I20240503 07:59:10.487633 9266 mvcc.cc:205] Tried to move back new op lower bound from 7023506024335159296 to 7023506023841316864. Current Snapshot: MvccSnapshot[applied={T|T < 7023506024335159296}]
I20240503 07:59:10.489109 9268 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:10.489748 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a] alter complete (version 1)
I20240503 07:59:10.489955 9268 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:10.495467 9269 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:10.496182 9269 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:10.498925 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a]). Expected version 1 got 0
I20240503 07:59:10.505875 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.513581 6408 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 07:59:10.565132 6561 consensus_queue.cc:237] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:10.566975 6296 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.567054 6428 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.567395 9278 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.567624 9277 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.569463 8893 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.569833 6296 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
W20240503 07:59:10.571817 6497 consensus_peers.cc:485] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e9ce52b9ae543cca72fb92efafc3c4d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.572731 6156 catalog_manager.cc:5484] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.573244 6428 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.579949 6428 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:10.581374 6561 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.581380 6691 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.581734 9288 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.581921 9289 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.583559 8895 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
W20240503 07:59:10.585752 6365 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a10085c34be4d31aae9f56a2ec1ba23. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.586401 6561 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.586510 6156 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.589340 6691 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.591526 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:10.595003 6561 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:10.596822 6691 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.596904 6428 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.597173 9277 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.599004 9277 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.600020 9114 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:10.600301 9114 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:10.600234 9277 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.601068 9269 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:10.601338 9269 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:10.601507 6691 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.602809 9268 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:10.603121 9268 tablet.cc:1620] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
W20240503 07:59:10.605654 6494 consensus_peers.cc:485] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c597bc1a19274f55b9d60558db35fcae. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.606071 6151 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.607197 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=9b45cc6bc4254577a20e6cdc2d8bd85a] alter complete (version 2)
I20240503 07:59:10.608753 6428 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.613814 6153 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 07:59:10.613929 6153 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 07:59:10.616042 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 0a144fda14a34046a06e558823f02735
I20240503 07:59:10.616322 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 7e10c6d976ed4a8ca0d68b56d7dc753f
I20240503 07:59:10.616566 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet bc0992eb4c7a471f9646518dd20e7c26
I20240503 07:59:10.616677 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a144fda14a34046a06e558823f02735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:10.616904 9178 tablet_replica.cc:330] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:10.616897 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a144fda14a34046a06e558823f02735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.617044 9178 raft_consensus.cc:2229] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.617210 9178 raft_consensus.cc:2258] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.617302 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:10.617460 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.617587 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.617666 9178 ts_tablet_manager.cc:1875] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.619499 9178 ts_tablet_manager.cc:1888] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.619547 9178 log.cc:1196] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0a144fda14a34046a06e558823f02735
I20240503 07:59:10.619740 9178 ts_tablet_manager.cc:1909] T 0a144fda14a34046a06e558823f02735 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:10.619908 9177 tablet_replica.cc:330] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.620035 9177 raft_consensus.cc:2229] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.620133 9177 raft_consensus.cc:2258] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.620565 9177 ts_tablet_manager.cc:1875] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.620857 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:10.621008 9176 tablet_replica.cc:330] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:10.621120 9176 raft_consensus.cc:2229] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.621210 9176 raft_consensus.cc:2258] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.621546 9176 ts_tablet_manager.cc:1875] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.622512 9177 ts_tablet_manager.cc:1888] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.622570 9177 log.cc:1196] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0a144fda14a34046a06e558823f02735
I20240503 07:59:10.622759 9177 ts_tablet_manager.cc:1909] T 0a144fda14a34046a06e558823f02735 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.623250 9177 tablet_replica.cc:330] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.623328 9177 raft_consensus.cc:2229] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.623396 9176 ts_tablet_manager.cc:1888] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.623407 9177 raft_consensus.cc:2258] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.623385 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 0a144fda14a34046a06e558823f02735 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.623451 9176 log.cc:1196] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7e10c6d976ed4a8ca0d68b56d7dc753f
I20240503 07:59:10.623679 9176 ts_tablet_manager.cc:1909] T 7e10c6d976ed4a8ca0d68b56d7dc753f P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:10.623747 9177 ts_tablet_manager.cc:1875] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.624198 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.624732 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0992eb4c7a471f9646518dd20e7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:10.624907 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e10c6d976ed4a8ca0d68b56d7dc753f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:10.624984 9178 tablet_replica.cc:330] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:10.624984 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 0a144fda14a34046a06e558823f02735 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.625051 9175 tablet_replica.cc:330] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:10.625121 9178 raft_consensus.cc:2229] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.625174 9175 raft_consensus.cc:2229] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.625226 9178 raft_consensus.cc:2258] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.625334 9177 ts_tablet_manager.cc:1888] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.625357 9175 raft_consensus.cc:2258] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.625403 9177 log.cc:1196] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/bc0992eb4c7a471f9646518dd20e7c26
I20240503 07:59:10.625640 9178 ts_tablet_manager.cc:1875] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.625814 9175 ts_tablet_manager.cc:1875] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.625839 9177 ts_tablet_manager.cc:1909] T bc0992eb4c7a471f9646518dd20e7c26 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.626336 9177 tablet_replica.cc:330] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.626454 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet bc0992eb4c7a471f9646518dd20e7c26 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.626864 9177 raft_consensus.cc:2229] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.627029 9177 raft_consensus.cc:2258] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.627763 9175 ts_tablet_manager.cc:1888] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.627828 9175 log.cc:1196] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/bc0992eb4c7a471f9646518dd20e7c26
I20240503 07:59:10.627862 9177 ts_tablet_manager.cc:1875] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.627864 6428 consensus_queue.cc:237] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:10.628044 9175 ts_tablet_manager.cc:1909] T bc0992eb4c7a471f9646518dd20e7c26 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:10.624728 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a144fda14a34046a06e558823f02735 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:10.628809 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet bc0992eb4c7a471f9646518dd20e7c26 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.629227 9175 tablet_replica.cc:330] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:10.629344 9175 raft_consensus.cc:2229] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.629474 9177 ts_tablet_manager.cc:1888] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.629527 9177 log.cc:1196] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e10c6d976ed4a8ca0d68b56d7dc753f
I20240503 07:59:10.629515 9175 raft_consensus.cc:2258] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.629642 6561 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.629736 9177 ts_tablet_manager.cc:1909] T 7e10c6d976ed4a8ca0d68b56d7dc753f P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.629900 9175 ts_tablet_manager.cc:1875] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.630276 9288 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.630353 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.631057 6296 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.631505 9175 ts_tablet_manager.cc:1888] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.631558 9175 log.cc:1196] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e10c6d976ed4a8ca0d68b56d7dc753f
I20240503 07:59:10.631726 9178 ts_tablet_manager.cc:1888] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.631772 9178 log.cc:1196] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc0992eb4c7a471f9646518dd20e7c26
I20240503 07:59:10.631769 9175 ts_tablet_manager.cc:1909] T 7e10c6d976ed4a8ca0d68b56d7dc753f P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:10.631985 9178 ts_tablet_manager.cc:1909] T bc0992eb4c7a471f9646518dd20e7c26 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:10.632169 9288 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.632278 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 7e10c6d976ed4a8ca0d68b56d7dc753f (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.632575 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet bc0992eb4c7a471f9646518dd20e7c26 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.632736 9175 tablet_replica.cc:330] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:10.632874 9175 raft_consensus.cc:2229] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 07:59:10.633002 6365 consensus_peers.cc:485] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 610160f6ba94483a9bdcf966c413e5d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.633026 9175 raft_consensus.cc:2258] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.633505 9175 ts_tablet_manager.cc:1875] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.634781 9175 ts_tablet_manager.cc:1888] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:10.634835 9175 log.cc:1196] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/0a144fda14a34046a06e558823f02735
I20240503 07:59:10.635006 9175 ts_tablet_manager.cc:1909] T 0a144fda14a34046a06e558823f02735 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:10.635541 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 0a144fda14a34046a06e558823f02735 (table rebalance_test_table_2 [id=6d629539a2874def9c039beeee9b9490]) successfully deleted
I20240503 07:59:10.635870 8895 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.636272 6561 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.638345 6296 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.639956 6156 catalog_manager.cc:5484] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.643057 9046 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.647326 6428 consensus_queue.cc:237] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:10.650467 6296 raft_consensus.cc:1261] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.650841 9289 consensus_queue.cc:1035] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.651293 9206 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.652236 6561 raft_consensus.cc:1261] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.652626 9289 consensus_queue.cc:1035] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:10.654022 6366 consensus_peers.cc:485] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e874c9cd82864973a92fb743e45ccf67. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.655081 9288 raft_consensus.cc:2932] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.655736 6296 raft_consensus.cc:2932] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.658066 6560 raft_consensus.cc:2932] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.659044 6157 catalog_manager.cc:5484] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.664569 6561 consensus_queue.cc:237] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:10.665750 6428 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.666025 6691 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.666330 9276 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.666548 9278 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.667902 9276 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.668370 6428 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
W20240503 07:59:10.669538 6496 consensus_peers.cc:485] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5538a709d8d440cb8f006a163f259389. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.669978 6691 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.670616 6153 catalog_manager.cc:5484] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.678036 6428 consensus_queue.cc:237] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:10.679388 6561 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.679415 6296 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.679701 9309 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.679903 9289 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.681308 9309 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.681699 6561 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.683509 6296 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
W20240503 07:59:10.684047 6363 consensus_peers.cc:485] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7e3555fb9214a40bd8dbf937b87cbaa. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.684015 6157 catalog_manager.cc:5484] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.691788 6691 consensus_queue.cc:237] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:10.693015 6561 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.693042 6296 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.693424 9347 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:10.693605 6627 consensus_peers.cc:485] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 065d5d0851e840939802896f9bddb84c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.693615 8296 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.695231 8077 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.695812 6561 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.695925 6296 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.698451 6151 catalog_manager.cc:5484] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.713898 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:10.718320 9114 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.719027 9269 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.719285 9268 tablet.cc:1722] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:10.722584 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=9b45cc6bc4254577a20e6cdc2d8bd85a] alter complete (version 3)
I20240503 07:59:10.784680 8956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.787309 8692 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:10.787974 8824 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:10.798241 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:10.801893 6671 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 07:59:10.808092 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:10.808313 6408 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:10.824672 6278 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:10.825094 6153 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:10.825222 6153 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:10.827297 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 7813cfc219a64d6f8d3678a053f82b5a
I20240503 07:59:10.827826 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:10.828104 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:10.828744 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 7813cfc219a64d6f8d3678a053f82b5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:10 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:10.828909 9177 tablet_replica.cc:330] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:10.829017 9177 raft_consensus.cc:2229] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.829106 9177 raft_consensus.cc:2258] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.829715 9177 ts_tablet_manager.cc:1875] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.831667 9177 ts_tablet_manager.cc:1888] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.831725 9177 log.cc:1196] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7813cfc219a64d6f8d3678a053f82b5a
I20240503 07:59:10.831952 9177 ts_tablet_manager.cc:1909] T 7813cfc219a64d6f8d3678a053f82b5a P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:10.836230 9189 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.837235 9398 tablet_replica.cc:330] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:10.837417 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table_new_name [id=9b45cc6bc4254577a20e6cdc2d8bd85a]) successfully deleted
I20240503 07:59:10.837524 9398 raft_consensus.cc:2229] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:10.837664 9398 raft_consensus.cc:2258] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.846107 9398 ts_tablet_manager.cc:1875] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.848129 9397 tablet_replica.cc:330] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:10.848309 9397 raft_consensus.cc:2229] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:10.848604 9397 raft_consensus.cc:2258] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:10.849433 9398 ts_tablet_manager.cc:1888] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.849561 9398 log.cc:1196] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/7813cfc219a64d6f8d3678a053f82b5a
I20240503 07:59:10.849907 9398 ts_tablet_manager.cc:1909] T 7813cfc219a64d6f8d3678a053f82b5a P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:10.850984 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table_new_name [id=9b45cc6bc4254577a20e6cdc2d8bd85a]) successfully deleted
I20240503 07:59:10.853279 9397 ts_tablet_manager.cc:1875] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:10.855597 9189 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.855942 9397 ts_tablet_manager.cc:1888] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:10.856020 9397 log.cc:1196] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7813cfc219a64d6f8d3678a053f82b5a
I20240503 07:59:10.856248 9397 ts_tablet_manager.cc:1909] T 7813cfc219a64d6f8d3678a053f82b5a P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:10.857177 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 7813cfc219a64d6f8d3678a053f82b5a (table rebalancer_dynamic_table_new_name [id=9b45cc6bc4254577a20e6cdc2d8bd85a]) successfully deleted
I20240503 07:59:10.886076 6428 consensus_queue.cc:237] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:10.887475 6561 raft_consensus.cc:1261] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.887560 6296 raft_consensus.cc:1261] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:10.887562 6363 consensus_peers.cc:485] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09c32a1e047844a3b09f2c790d735c9b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.887856 9288 consensus_queue.cc:1035] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.888046 9309 consensus_queue.cc:1035] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.889783 9289 raft_consensus.cc:2932] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.890247 6296 raft_consensus.cc:2932] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.890247 6561 raft_consensus.cc:2932] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.892796 6153 catalog_manager.cc:5484] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.899969 6428 consensus_queue.cc:237] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:10.901201 6691 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:10.901580 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8384cbf3f368424d9f8a9060efe9c0f2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.901624 9309 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.901670 6561 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.902021 9309 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.903451 8895 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.903786 6690 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.904258 6561 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.906030 6149 catalog_manager.cc:5484] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.913061 6428 consensus_queue.cc:237] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:10.914177 6561 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.914222 6296 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:10.914417 6365 consensus_peers.cc:485] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 93eb2db48ef449a486a0347a76a8a5ea. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.914526 9288 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.914816 9309 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.916148 9288 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.916528 6561 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.916967 6296 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.919445 6154 catalog_manager.cc:5484] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.926828 6690 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:10.928068 6428 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:10.928285 6627 consensus_peers.cc:485] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82de209205b342cfaac10506fdac9303. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.928315 6561 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.928421 8077 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.928699 9347 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.929680 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:10.930047 8077 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.930419 6428 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.932689 6154 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.934510 6561 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:10.935674 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]), partition=
I20240503 07:59:10.935736 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]), partition=
I20240503 07:59:10.935910 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442bf6b34d6b48c29fc497372128eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.935987 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442bf6b34d6b48c29fc497372128eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.936134 6671 tablet_service.cc:1467] Processing CreateTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]), partition=
I20240503 07:59:10.936321 6671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442bf6b34d6b48c29fc497372128eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.938256 9440 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:10.938287 9441 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:10.938625 9347 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.939117 9441 tablet_bootstrap.cc:654] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.939121 9440 tablet_bootstrap.cc:654] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.940057 9440 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:10.940174 9440 ts_tablet_manager.cc:1367] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:10.940620 9440 raft_consensus.cc:348] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.940721 9440 raft_consensus.cc:374] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.940917 9440 raft_consensus.cc:729] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.941076 9440 consensus_queue.cc:260] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.941409 9440 ts_tablet_manager.cc:1398] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.942142 9439 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:10.943133 9439 tablet_bootstrap.cc:654] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:10.943869 8296 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:10.944063 9441 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:10.944139 9441 ts_tablet_manager.cc:1367] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:10.944592 9441 raft_consensus.cc:348] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.944679 9441 raft_consensus.cc:374] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.944708 9441 raft_consensus.cc:729] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.945315 9441 consensus_queue.cc:260] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.945777 9441 ts_tablet_manager.cc:1398] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:10.946977 9439 tablet_bootstrap.cc:492] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:10.947100 9439 ts_tablet_manager.cc:1367] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:10.947516 9439 raft_consensus.cc:348] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.947657 9439 raft_consensus.cc:374] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:10.947708 9439 raft_consensus.cc:729] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:10.947840 9439 consensus_queue.cc:260] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:10.948156 9439 ts_tablet_manager.cc:1398] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:10.950318 6690 consensus_queue.cc:237] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:10.951707 6428 raft_consensus.cc:1261] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.952139 6296 raft_consensus.cc:1261] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.952479 8296 consensus_queue.cc:1035] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.952654 8077 consensus_queue.cc:1035] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.954247 8296 raft_consensus.cc:2932] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.954627 6296 raft_consensus.cc:2932] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.956542 6428 raft_consensus.cc:2932] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:10.957016 6149 catalog_manager.cc:5484] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:10.957906 6625 consensus_peers.cc:485] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5710dff79184444797bb46d981ae6f55. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.961342 9452 ts_tablet_manager.cc:902] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:10.962487 9452 tablet_copy_client.cc:308] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:10.967442 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 0e9ce52b9ae543cca72fb92efafc3c4d from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:43151)
I20240503 07:59:10.967550 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 0e9ce52b9ae543cca72fb92efafc3c4d from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:43151: session id = 9553fe3c4ccd4e91b7c176aad82fec09-0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:10.967906 6296 consensus_queue.cc:237] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:10.968636 6581 tablet_copy_source_session.cc:205] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:10.969412 6690 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.969691 6428 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.970191 9465 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.970619 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9ce52b9ae543cca72fb92efafc3c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:10.970606 9465 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.972255 9240 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.973380 9452 tablet_copy_client.cc:783] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:10.973671 9452 tablet_copy_client.cc:647] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:10.974674 6690 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.975461 9452 tablet_copy_client.cc:523] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 07:59:10.975920 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f417c61594348428ee9514ded49ee88. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.976642 6154 catalog_manager.cc:5484] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:10.977474 6428 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:10.977679 9452 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:10.985110 6296 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:10.986316 6428 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.986476 6561 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:10.986743 9464 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.986975 9470 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:10.989307 9464 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.991422 6560 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:10.993523 6149 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:10.995956 6234 consensus_peers.cc:485] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f943ecfe59f432c861a54b16c62572f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:10.997339 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:11.000375 6428 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:11.005781 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 551f25dcee034b7894498e3c748b0f0e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.005867 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 551f25dcee034b7894498e3c748b0f0e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.006022 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551f25dcee034b7894498e3c748b0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.006096 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551f25dcee034b7894498e3c748b0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.006482 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 551f25dcee034b7894498e3c748b0f0e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.006705 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551f25dcee034b7894498e3c748b0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.007416 9486 ts_tablet_manager.cc:902] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.008749 9440 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.008913 9486 tablet_copy_client.cc:308] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.009574 9440 tablet_bootstrap.cc:654] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.009583 8957 tablet_service.cc:1467] Processing CreateTablet for tablet f534eefb7c334889a9c5d02175ae2d94 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.009824 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f534eefb7c334889a9c5d02175ae2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.009863 8692 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa99be75f498a8394aabbfabb3f5a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.010068 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa99be75f498a8394aabbfabb3f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.008697 8691 tablet_service.cc:1467] Processing CreateTablet for tablet f534eefb7c334889a9c5d02175ae2d94 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.010710 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f534eefb7c334889a9c5d02175ae2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.010859 9440 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.010937 9440 ts_tablet_manager.cc:1367] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:11.011339 9440 raft_consensus.cc:348] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.011427 9440 raft_consensus.cc:374] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.011458 9440 raft_consensus.cc:729] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.011561 9440 consensus_queue.cc:260] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.011796 9440 ts_tablet_manager.cc:1398] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:11.011955 9440 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.012652 9439 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.012676 8561 tablet_service.cc:1467] Processing CreateTablet for tablet f534eefb7c334889a9c5d02175ae2d94 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.009563 8956 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa99be75f498a8394aabbfabb3f5a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.012926 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f534eefb7c334889a9c5d02175ae2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.012964 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa99be75f498a8394aabbfabb3f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.013579 9439 tablet_bootstrap.cc:654] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.015172 8560 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa99be75f498a8394aabbfabb3f5a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.015394 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa99be75f498a8394aabbfabb3f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.017441 9440 tablet_bootstrap.cc:654] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.017665 9452 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.017886 9491 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.018946 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 610160f6ba94483a9bdcf966c413e5d1 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:58189)
I20240503 07:59:11.019054 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 610160f6ba94483a9bdcf966c413e5d1 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:58189: session id = 616fdda6ba714cb7be01a564de2b9db2-610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:11.020344 6450 tablet_copy_source_session.cc:205] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.021126 9491 tablet_bootstrap.cc:654] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.021459 9439 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:11.021543 9439 ts_tablet_manager.cc:1367] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:11.021773 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610160f6ba94483a9bdcf966c413e5d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.021920 9452 tablet_bootstrap.cc:492] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:11.022073 9452 ts_tablet_manager.cc:1367] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.044s user 0.029s sys 0.000s
I20240503 07:59:11.021984 9439 raft_consensus.cc:348] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.022118 9439 raft_consensus.cc:374] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.022148 9439 raft_consensus.cc:729] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.022261 9439 consensus_queue.cc:260] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.022418 9452 raft_consensus.cc:348] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.022527 9452 raft_consensus.cc:729] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:11.022544 9439 ts_tablet_manager.cc:1398] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.022619 9452 consensus_queue.cc:260] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.022696 9439 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.022863 9452 ts_tablet_manager.cc:1398] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.023461 9487 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:11.023885 9486 tablet_copy_client.cc:783] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.024065 9486 tablet_copy_client.cc:647] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.024951 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-0e9ce52b9ae543cca72fb92efafc3c4d received from {username='slave'} at 127.0.99.134:43151
I20240503 07:59:11.025044 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-0e9ce52b9ae543cca72fb92efafc3c4d on tablet 0e9ce52b9ae543cca72fb92efafc3c4d with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.025677 9491 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:11.025806 9491 ts_tablet_manager.cc:1367] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:11.026518 9491 raft_consensus.cc:348] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.026670 9486 tablet_copy_client.cc:523] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.026763 9485 ts_tablet_manager.cc:902] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:11.027980 9487 tablet_bootstrap.cc:654] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.028017 9486 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.028734 9485 tablet_copy_client.cc:308] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:11.030468 9439 tablet_bootstrap.cc:654] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.032145 6560 consensus_queue.cc:237] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.032660 9491 raft_consensus.cc:374] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.032763 9491 raft_consensus.cc:729] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.032961 9491 consensus_queue.cc:260] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.034309 6296 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.034530 6428 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.034665 9277 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.034914 8893 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.037264 9440 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.037478 9440 ts_tablet_manager.cc:1367] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.002s
W20240503 07:59:11.037505 6496 consensus_peers.cc:485] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d95cc5335534f02aa62b00fb8b9c33c. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.037886 9440 raft_consensus.cc:348] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.037971 9440 raft_consensus.cc:374] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.037998 9440 raft_consensus.cc:729] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.038166 9440 consensus_queue.cc:260] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.038825 9440 ts_tablet_manager.cc:1398] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:11.039023 9440 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.040331 9440 tablet_bootstrap.cc:654] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.044973 9491 ts_tablet_manager.cc:1398] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20240503 07:59:11.045552 9491 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.045819 9277 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.046248 6428 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.046445 9491 tablet_bootstrap.cc:654] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.047403 9400 raft_consensus.cc:482] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.047477 9487 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:11.047477 9400 raft_consensus.cc:504] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.047514 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet c597bc1a19274f55b9d60558db35fcae from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:60775)
I20240503 07:59:11.047571 9487 ts_tablet_manager.cc:1367] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 07:59:11.047585 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet c597bc1a19274f55b9d60558db35fcae from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:60775: session id = c8d71787690245e681c85a087086e7ed-c597bc1a19274f55b9d60558db35fcae
I20240503 07:59:11.047878 9400 leader_election.cc:288] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.048015 9487 raft_consensus.cc:348] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.048125 9487 raft_consensus.cc:374] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.048440 6581 tablet_copy_source_session.cc:205] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.048156 9487 raft_consensus.cc:729] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.048873 9487 consensus_queue.cc:260] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.049212 9487 ts_tablet_manager.cc:1398] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.050479 6149 catalog_manager.cc:5484] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.052788 6296 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.053177 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:11.053654 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c597bc1a19274f55b9d60558db35fcae. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:11.053745 8647 leader_election.cc:341] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:11.057044 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:11.057976 8645 leader_election.cc:341] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:11.058054 8645 leader_election.cc:302] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09; no voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.058260 9400 raft_consensus.cc:2726] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:11.059106 9485 tablet_copy_client.cc:783] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.059305 9485 tablet_copy_client.cc:647] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.060540 9485 tablet_copy_client.cc:523] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.060933 9439 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:11.061024 9439 ts_tablet_manager.cc:1367] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.000s
I20240503 07:59:11.061419 9439 raft_consensus.cc:348] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.061522 9439 raft_consensus.cc:374] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.061559 9439 raft_consensus.cc:729] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.061664 9439 consensus_queue.cc:260] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.061916 9439 ts_tablet_manager.cc:1398] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.062013 9485 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.063387 9511 ts_tablet_manager.cc:902] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.064477 9511 tablet_copy_client.cc:308] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.065881 9491 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:11.065954 9491 ts_tablet_manager.cc:1367] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:11.066304 9491 raft_consensus.cc:348] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.066394 9491 raft_consensus.cc:374] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.066421 9491 raft_consensus.cc:729] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.066519 9491 consensus_queue.cc:260] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.066718 9491 ts_tablet_manager.cc:1398] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.066834 9491 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.067613 9491 tablet_bootstrap.cc:654] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.068580 9400 raft_consensus.cc:482] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.068658 9400 raft_consensus.cc:504] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:11.068746 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 2a10085c34be4d31aae9f56a2ec1ba23 from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:53021)
I20240503 07:59:11.068853 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 2a10085c34be4d31aae9f56a2ec1ba23 from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:53021: session id = 3d08a06d53b14381bee6223c4e1a6027-2a10085c34be4d31aae9f56a2ec1ba23
I20240503 07:59:11.069059 9400 leader_election.cc:288] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:11.069607 9440 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.069687 9440 ts_tablet_manager.cc:1367] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20240503 07:59:11.069689 6450 tablet_copy_source_session.cc:205] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.070098 9440 raft_consensus.cc:348] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.070189 9440 raft_consensus.cc:374] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.070219 9440 raft_consensus.cc:729] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.070338 9440 consensus_queue.cc:260] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.070356 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442bf6b34d6b48c29fc497372128eaf0" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:11.070448 8977 raft_consensus.cc:2454] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:11.070576 9440 ts_tablet_manager.cc:1398] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:11.070708 8647 leader_election.cc:302] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:11.070874 9189 raft_consensus.cc:2781] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.070919 9189 raft_consensus.cc:482] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.070952 9189 raft_consensus.cc:3037] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.071619 9189 raft_consensus.cc:504] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:11.071897 9189 leader_election.cc:288] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:11.072978 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442bf6b34d6b48c29fc497372128eaf0" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:11.073078 8977 raft_consensus.cc:3037] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.073137 9491 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:11.073202 9491 ts_tablet_manager.cc:1367] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:11.073649 9491 raft_consensus.cc:348] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.073724 9491 raft_consensus.cc:374] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.073722 8977 raft_consensus.cc:2454] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.073750 9491 raft_consensus.cc:729] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.073837 9491 consensus_queue.cc:260] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.073990 8647 leader_election.cc:302] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:11.074031 9491 ts_tablet_manager.cc:1398] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.074702 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a10085c34be4d31aae9f56a2ec1ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.074687 9189 raft_consensus.cc:2781] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.074760 9189 raft_consensus.cc:686] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:11.074862 9189 consensus_queue.cc:237] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:11.078378 9511 tablet_copy_client.cc:783] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.078593 9511 tablet_copy_client.cc:647] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.078794 6153 catalog_manager.cc:5484] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.080909 9511 tablet_copy_client.cc:523] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.082520 9511 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.085832 6690 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.087216 6296 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:11.087659 6625 consensus_peers.cc:485] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39b01318dc1543cba2cc991e9db91c28. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.087786 8077 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.088331 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442bf6b34d6b48c29fc497372128eaf0" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:11.088419 6691 raft_consensus.cc:3037] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.088064 6560 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.089334 6691 raft_consensus.cc:2454] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.089360 9486 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.089612 9486 tablet_bootstrap.cc:492] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:11.089637 8077 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.090142 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442bf6b34d6b48c29fc497372128eaf0" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:11.090241 6690 raft_consensus.cc:2362] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1. Re-sending same reply.
I20240503 07:59:11.091392 9347 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:11.091760 6560 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:11.092978 9486 ts_tablet_manager.cc:1367] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.065s user 0.022s sys 0.008s
I20240503 07:59:11.093437 9486 raft_consensus.cc:348] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.093583 9486 raft_consensus.cc:729] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:11.093708 9486 consensus_queue.cc:260] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.095168 9486 ts_tablet_manager.cc:1398] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:11.095067 6154 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:11.095714 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-610160f6ba94483a9bdcf966c413e5d1 received from {username='slave'} at 127.0.99.135:58189
I20240503 07:59:11.095803 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-610160f6ba94483a9bdcf966c413e5d1 on tablet 610160f6ba94483a9bdcf966c413e5d1 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:11.099293 6296 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:11.105409 6690 consensus_queue.cc:237] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.106763 6428 raft_consensus.cc:1261] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.107234 8296 consensus_queue.cc:1035] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.107877 6296 raft_consensus.cc:1261] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.108424 8077 consensus_queue.cc:1035] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.110504 9485 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.110824 9485 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:11.111012 9485 ts_tablet_manager.cc:1367] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.049s user 0.020s sys 0.007s
I20240503 07:59:11.111465 9485 raft_consensus.cc:348] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.111613 9485 raft_consensus.cc:729] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:11.111727 9485 consensus_queue.cc:260] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
W20240503 07:59:11.111750 6628 consensus_peers.cc:485] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd67fb07203e4e99919ec0317bdf077d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.112047 9485 ts_tablet_manager.cc:1398] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.112524 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-c597bc1a19274f55b9d60558db35fcae received from {username='slave'} at 127.0.99.133:60775
I20240503 07:59:11.112604 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session c8d71787690245e681c85a087086e7ed-c597bc1a19274f55b9d60558db35fcae on tablet c597bc1a19274f55b9d60558db35fcae with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.113425 8077 raft_consensus.cc:2932] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.115936 6428 raft_consensus.cc:2932] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.117635 6157 catalog_manager.cc:5484] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.118657 6149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 07:59:11.118763 6149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 07:59:11.118948 9538 ts_tablet_manager.cc:902] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:11.119436 9538 tablet_copy_client.cc:308] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:11.121675 6296 raft_consensus.cc:2932] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.122313 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet e874c9cd82864973a92fb743e45ccf67
I20240503 07:59:11.122629 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 293067841cc6412f853e4da6df8f6b36
I20240503 07:59:11.122834 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet e874c9cd82864973a92fb743e45ccf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:11.122855 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 86ba39eb4cb245e5bc9c8bea6b1acb09
I20240503 07:59:11.122985 9176 tablet_replica.cc:330] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:11.123108 9176 raft_consensus.cc:2229] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.123284 9176 raft_consensus.cc:2258] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.123644 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet e874c9cd82864973a92fb743e45ccf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:11.123812 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 293067841cc6412f853e4da6df8f6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:11.124012 9176 ts_tablet_manager.cc:1875] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.124243 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:11.124415 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet e874c9cd82864973a92fb743e45ccf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:11.124590 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.125146 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.125311 9175 tablet_replica.cc:330] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.125340 9177 tablet_replica.cc:330] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.125443 9175 raft_consensus.cc:2229] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.125465 9177 raft_consensus.cc:2229] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.125547 9175 raft_consensus.cc:2258] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.125622 9177 raft_consensus.cc:2258] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.125990 9175 ts_tablet_manager.cc:1875] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.126070 9177 ts_tablet_manager.cc:1875] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.126116 9176 ts_tablet_manager.cc:1888] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.126168 9176 log.cc:1196] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e874c9cd82864973a92fb743e45ccf67
I20240503 07:59:11.126389 9176 ts_tablet_manager.cc:1909] T e874c9cd82864973a92fb743e45ccf67 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:11.126868 9176 tablet_replica.cc:330] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:11.126992 9176 raft_consensus.cc:2229] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.127140 9178 tablet_replica.cc:330] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:11.127251 9178 raft_consensus.cc:2229] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.127480 9178 raft_consensus.cc:2258] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.127748 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet 293067841cc6412f853e4da6df8f6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.127893 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet e874c9cd82864973a92fb743e45ccf67 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.127943 9175 ts_tablet_manager.cc:1888] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.128001 9175 log.cc:1196] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/86ba39eb4cb245e5bc9c8bea6b1acb09
I20240503 07:59:11.128018 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 293067841cc6412f853e4da6df8f6b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.128054 9177 ts_tablet_manager.cc:1888] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.128103 9177 log.cc:1196] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/86ba39eb4cb245e5bc9c8bea6b1acb09
I20240503 07:59:11.128242 9175 ts_tablet_manager.cc:1909] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.128348 9177 ts_tablet_manager.cc:1909] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.128696 9175 tablet_replica.cc:330] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.129061 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet e874c9cd82864973a92fb743e45ccf67 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
W20240503 07:59:11.129104 6144 catalog_manager.cc:4797] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): delete failed for tablet e874c9cd82864973a92fb743e45ccf67 because the tablet was not found. No further retry: Not found: Tablet not found: e874c9cd82864973a92fb743e45ccf67
I20240503 07:59:11.129181 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.129268 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.129388 9177 tablet_replica.cc:330] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.129494 9177 raft_consensus.cc:2229] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.129652 9177 raft_consensus.cc:2258] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.130162 9178 ts_tablet_manager.cc:1875] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.130620 6428 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.131378 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 5538a709d8d440cb8f006a163f259389 from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:36739)
I20240503 07:59:11.131457 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 5538a709d8d440cb8f006a163f259389 from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:36739: session id = 3d08a06d53b14381bee6223c4e1a6027-5538a709d8d440cb8f006a163f259389
I20240503 07:59:11.132126 9178 ts_tablet_manager.cc:1888] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.132194 9178 log.cc:1196] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e874c9cd82864973a92fb743e45ccf67
I20240503 07:59:11.132258 6581 tablet_copy_source_session.cc:205] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 07:59:11.132354 6365 consensus_peers.cc:485] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5631f3e97fe04013a2a7140f44260003. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.132441 9178 ts_tablet_manager.cc:1909] T e874c9cd82864973a92fb743e45ccf67 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:11.132642 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.133018 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet e874c9cd82864973a92fb743e45ccf67 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.132947 9511 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.133147 9289 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.133410 9511 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:11.133673 9511 ts_tablet_manager.cc:1367] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.006s
I20240503 07:59:11.133687 6560 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.134241 9289 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.134151 9511 raft_consensus.cc:348] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.134481 9511 raft_consensus.cc:729] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:11.134574 9177 ts_tablet_manager.cc:1875] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.134693 9511 consensus_queue.cc:260] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.135051 9175 raft_consensus.cc:2229] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.135253 9175 raft_consensus.cc:2258] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.135613 9511 ts_tablet_manager.cc:1398] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.135746 9175 ts_tablet_manager.cc:1875] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.135890 9289 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:11.136077 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-2a10085c34be4d31aae9f56a2ec1ba23 received from {username='slave'} at 127.0.99.136:53021
I20240503 07:59:11.136150 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-2a10085c34be4d31aae9f56a2ec1ba23 on tablet 2a10085c34be4d31aae9f56a2ec1ba23 with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:11.136262 9177 ts_tablet_manager.cc:1888] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.136333 9177 log.cc:1196] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e874c9cd82864973a92fb743e45ccf67
I20240503 07:59:11.136554 9177 ts_tablet_manager.cc:1909] T e874c9cd82864973a92fb743e45ccf67 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.137179 6560 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:11.137451 9176 raft_consensus.cc:2258] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.137769 9175 ts_tablet_manager.cc:1888] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.137823 9175 log.cc:1196] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/293067841cc6412f853e4da6df8f6b36
I20240503 07:59:11.137893 9176 ts_tablet_manager.cc:1875] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.138090 9175 ts_tablet_manager.cc:1909] T 293067841cc6412f853e4da6df8f6b36 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.138983 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5538a709d8d440cb8f006a163f259389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.139102 9177 tablet_replica.cc:330] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.139111 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 293067841cc6412f853e4da6df8f6b36 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.139212 9177 raft_consensus.cc:2229] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.139254 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet e874c9cd82864973a92fb743e45ccf67 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.139317 9177 raft_consensus.cc:2258] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.139710 9177 ts_tablet_manager.cc:1875] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.139964 6149 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.141038 9538 tablet_copy_client.cc:783] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.141216 9538 tablet_copy_client.cc:647] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.141387 9177 ts_tablet_manager.cc:1888] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.141461 9177 log.cc:1196] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/293067841cc6412f853e4da6df8f6b36
I20240503 07:59:11.141696 9177 ts_tablet_manager.cc:1909] T 293067841cc6412f853e4da6df8f6b36 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.142326 9538 tablet_copy_client.cc:523] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.142643 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 293067841cc6412f853e4da6df8f6b36 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.142946 9178 tablet_replica.cc:330] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:11.143218 9178 raft_consensus.cc:2229] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.143388 9178 raft_consensus.cc:2258] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.143446 9538 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.143376 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:11.143880 9178 ts_tablet_manager.cc:1875] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.146375 9178 ts_tablet_manager.cc:1888] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.146948 9176 ts_tablet_manager.cc:1888] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.147043 9176 log.cc:1196] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/86ba39eb4cb245e5bc9c8bea6b1acb09
I20240503 07:59:11.147296 9176 ts_tablet_manager.cc:1909] T 86ba39eb4cb245e5bc9c8bea6b1acb09 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:11.147410 9178 log.cc:1196] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/293067841cc6412f853e4da6df8f6b36
I20240503 07:59:11.147671 9178 ts_tablet_manager.cc:1909] T 293067841cc6412f853e4da6df8f6b36 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:11.147960 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 86ba39eb4cb245e5bc9c8bea6b1acb09 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.148326 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 293067841cc6412f853e4da6df8f6b36 (table rebalance_test_table_4 [id=cc23875e4c9c4380bd7862acd33adc2b]) successfully deleted
I20240503 07:59:11.152835 6296 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.154263 6428 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.154675 9465 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.156373 9470 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.156982 6691 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.157331 9240 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.159473 6428 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
W20240503 07:59:11.160241 6234 consensus_peers.cc:485] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4308bc87be24f28b6566ce76cae6e50. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.160910 6691 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:11.161178 6157 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.176170 9538 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.176528 9538 tablet_bootstrap.cc:492] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:11.176712 9538 ts_tablet_manager.cc:1367] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.004s
I20240503 07:59:11.177102 9538 raft_consensus.cc:348] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.177268 9538 raft_consensus.cc:729] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:11.177371 9538 consensus_queue.cc:260] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.177747 9538 ts_tablet_manager.cc:1398] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.178161 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-5538a709d8d440cb8f006a163f259389 received from {username='slave'} at 127.0.99.136:36739
I20240503 07:59:11.178231 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-5538a709d8d440cb8f006a163f259389 on tablet 5538a709d8d440cb8f006a163f259389 with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:11.180382 9196 raft_consensus.cc:482] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.180464 9196 raft_consensus.cc:504] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.181485 9196 leader_election.cc:288] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.185994 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:11.186121 8575 raft_consensus.cc:2454] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:11.186497 8910 leader_election.cc:302] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:11.186731 9196 raft_consensus.cc:2781] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.186800 9196 raft_consensus.cc:482] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.186833 9196 raft_consensus.cc:3037] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.187680 9196 raft_consensus.cc:504] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.187701 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:11.187826 8712 raft_consensus.cc:2454] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:11.187958 9196 leader_election.cc:288] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.188740 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:11.188854 8712 raft_consensus.cc:3037] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.189811 8712 raft_consensus.cc:2454] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:11.189883 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa99be75f498a8394aabbfabb3f5a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:11.189982 8575 raft_consensus.cc:3037] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.190088 8913 leader_election.cc:302] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:11.190369 9196 raft_consensus.cc:2781] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.190515 9196 raft_consensus.cc:686] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:11.190668 9486 ts_tablet_manager.cc:902] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:11.190748 9196 consensus_queue.cc:237] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.190801 8575 raft_consensus.cc:2454] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:11.191083 9486 tablet_copy_client.cc:308] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:11.192755 6149 catalog_manager.cc:5484] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.195598 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 065d5d0851e840939802896f9bddb84c from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:11.195699 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 065d5d0851e840939802896f9bddb84c from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-065d5d0851e840939802896f9bddb84c
I20240503 07:59:11.196866 6712 tablet_copy_source_session.cc:205] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.197886 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065d5d0851e840939802896f9bddb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.199723 9486 tablet_copy_client.cc:783] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.199854 9486 tablet_copy_client.cc:647] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.201160 9486 tablet_copy_client.cc:523] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.202106 9486 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.211350 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:11.215549 8977 raft_consensus.cc:1261] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:11.215858 6691 raft_consensus.cc:1261] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:11.216424 9189 consensus_queue.cc:1035] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.216691 9189 consensus_queue.cc:1035] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.220288 6149 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.222038 9400 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.222116 9400 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.222437 9400 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.222851 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:11.223009 8575 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:11.223023 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:11.223117 8976 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:11.223536 8645 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:11.223702 9400 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.223737 6149 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.223764 9400 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.223802 9400 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.224118 6154 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.224706 9400 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.225057 9400 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.225545 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:11.225649 8575 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.226385 6149 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.226483 8575 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.226527 9486 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.226739 9486 tablet_bootstrap.cc:492] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:11.226846 8645 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:11.226979 9486 ts_tablet_manager.cc:1367] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20240503 07:59:11.227170 9248 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.227267 9248 raft_consensus.cc:686] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:11.227407 9248 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.227458 6149 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.227427 9486 raft_consensus.cc:348] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.227528 9486 raft_consensus.cc:729] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:11.227643 9486 consensus_queue.cc:260] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.227941 9486 ts_tablet_manager.cc:1398] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.228406 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-065d5d0851e840939802896f9bddb84c received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:11.228480 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-065d5d0851e840939802896f9bddb84c on tablet 065d5d0851e840939802896f9bddb84c with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:11.228842 6154 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:11.229648 9613 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.230540 9613 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:11.231349 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:11.231473 8977 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.232255 8977 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.233587 9611 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.234190 9611 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:11.234586 9610 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:11.234849 9610 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:11.239723 6149 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a]). Expected version 1 got 0
I20240503 07:59:11.240160 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a] alter complete (version 1)
I20240503 07:59:11.254639 9485 ts_tablet_manager.cc:902] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.255009 9485 tablet_copy_client.cc:308] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.255403 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet c7e3555fb9214a40bd8dbf937b87cbaa from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:56365)
I20240503 07:59:11.255479 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet c7e3555fb9214a40bd8dbf937b87cbaa from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:56365: session id = c8d71787690245e681c85a087086e7ed-c7e3555fb9214a40bd8dbf937b87cbaa
I20240503 07:59:11.256287 6450 tablet_copy_source_session.cc:205] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.257033 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e3555fb9214a40bd8dbf937b87cbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.259055 9485 tablet_copy_client.cc:783] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.259193 9485 tablet_copy_client.cc:647] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.260273 9485 tablet_copy_client.cc:523] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.261425 9485 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.284111 9485 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.284346 9485 tablet_bootstrap.cc:492] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:11.284551 9485 ts_tablet_manager.cc:1367] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20240503 07:59:11.284999 9485 raft_consensus.cc:348] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.285135 9485 raft_consensus.cc:729] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:11.285238 9485 consensus_queue.cc:260] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.285614 9485 ts_tablet_manager.cc:1398] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.286113 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-c7e3555fb9214a40bd8dbf937b87cbaa received from {username='slave'} at 127.0.99.133:56365
I20240503 07:59:11.286180 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session c8d71787690245e681c85a087086e7ed-c7e3555fb9214a40bd8dbf937b87cbaa on tablet c7e3555fb9214a40bd8dbf937b87cbaa with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.298832 9464 raft_consensus.cc:482] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.298921 9464 raft_consensus.cc:504] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.299331 9464 leader_election.cc:288] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.299654 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551f25dcee034b7894498e3c748b0f0e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:11.299784 8712 raft_consensus.cc:2454] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:11.300336 6235 leader_election.cc:302] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:11.300850 9464 raft_consensus.cc:2781] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.300909 9464 raft_consensus.cc:482] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.300944 9464 raft_consensus.cc:3037] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.301731 9464 raft_consensus.cc:504] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.302068 9464 leader_election.cc:288] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:11.302471 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551f25dcee034b7894498e3c748b0f0e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:11.302603 8712 raft_consensus.cc:3037] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.303567 8712 raft_consensus.cc:2454] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:11.303915 6235 leader_election.cc:302] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:11.304749 9464 raft_consensus.cc:2781] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.304872 9464 raft_consensus.cc:686] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:11.305006 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551f25dcee034b7894498e3c748b0f0e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:11.304999 9464 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:11.305027 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551f25dcee034b7894498e3c748b0f0e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:11.305125 8575 raft_consensus.cc:2454] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:11.307039 6149 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.322057 9538 ts_tablet_manager.cc:902] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.322538 9538 tablet_copy_client.cc:308] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.323078 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 93eb2db48ef449a486a0347a76a8a5ea from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:53021)
I20240503 07:59:11.323177 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 93eb2db48ef449a486a0347a76a8a5ea from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:53021: session id = 3d08a06d53b14381bee6223c4e1a6027-93eb2db48ef449a486a0347a76a8a5ea
I20240503 07:59:11.324103 6450 tablet_copy_source_session.cc:205] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.325037 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93eb2db48ef449a486a0347a76a8a5ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.326854 9538 tablet_copy_client.cc:783] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.326980 9538 tablet_copy_client.cc:647] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.327838 9485 ts_tablet_manager.cc:902] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:11.328263 9485 tablet_copy_client.cc:308] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:11.329303 9538 tablet_copy_client.cc:523] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.330268 9538 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.332129 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 5710dff79184444797bb46d981ae6f55 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:54869)
I20240503 07:59:11.332211 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 5710dff79184444797bb46d981ae6f55 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:54869: session id = c8d71787690245e681c85a087086e7ed-5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.333062 6712 tablet_copy_source_session.cc:205] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.333935 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5710dff79184444797bb46d981ae6f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.335827 9485 tablet_copy_client.cc:783] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.335963 9485 tablet_copy_client.cc:647] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.342437 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:11.349972 9610 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.350250 9610 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:11.351457 9538 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.351696 9538 tablet_bootstrap.cc:492] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:11.351919 9611 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.352145 9538 ts_tablet_manager.cc:1367] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 07:59:11.352195 9611 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:11.352180 9613 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:11.352442 9613 tablet.cc:1620] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:11.352555 9538 raft_consensus.cc:348] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.352645 9538 raft_consensus.cc:729] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:11.352733 9538 consensus_queue.cc:260] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.353519 9485 tablet_copy_client.cc:523] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.354665 9485 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.356875 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=6ce0d065dc334ed5a9c8b1577ef35b3a] alter complete (version 2)
I20240503 07:59:11.358517 9538 ts_tablet_manager.cc:1398] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:11.359042 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-93eb2db48ef449a486a0347a76a8a5ea received from {username='slave'} at 127.0.99.136:53021
I20240503 07:59:11.359114 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-93eb2db48ef449a486a0347a76a8a5ea on tablet 93eb2db48ef449a486a0347a76a8a5ea with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:11.380810 9485 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.381054 9485 tablet_bootstrap.cc:492] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:11.381219 9485 ts_tablet_manager.cc:1367] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.001s
I20240503 07:59:11.382062 9485 raft_consensus.cc:348] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.382176 9485 raft_consensus.cc:729] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:11.382272 9485 consensus_queue.cc:260] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.382540 9485 ts_tablet_manager.cc:1398] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.382992 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-5710dff79184444797bb46d981ae6f55 received from {username='slave'} at 127.0.99.133:54869
I20240503 07:59:11.383059 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session c8d71787690245e681c85a087086e7ed-5710dff79184444797bb46d981ae6f55 on tablet 5710dff79184444797bb46d981ae6f55 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.412510 9485 ts_tablet_manager.cc:902] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.414199 9485 tablet_copy_client.cc:308] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.414757 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 09c32a1e047844a3b09f2c790d735c9b from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:56365)
I20240503 07:59:11.414849 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 09c32a1e047844a3b09f2c790d735c9b from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:56365: session id = c8d71787690245e681c85a087086e7ed-09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:11.415764 6450 tablet_copy_source_session.cc:205] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.418524 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c32a1e047844a3b09f2c790d735c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.419713 8956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.419907 8691 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:11.421151 8824 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:11.428215 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:11.438480 6279 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:11.438589 6410 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:11.439306 6671 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 07:59:11.441901 9485 tablet_copy_client.cc:783] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.441984 9452 ts_tablet_manager.cc:902] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.442049 9485 tablet_copy_client.cc:647] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.442430 9452 tablet_copy_client.cc:308] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.443168 9485 tablet_copy_client.cc:523] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.444236 9485 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.453122 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 8384cbf3f368424d9f8a9060efe9c0f2 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:40145)
I20240503 07:59:11.453217 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 8384cbf3f368424d9f8a9060efe9c0f2 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:40145: session id = 9553fe3c4ccd4e91b7c176aad82fec09-8384cbf3f368424d9f8a9060efe9c0f2
I20240503 07:59:11.454030 6450 tablet_copy_source_session.cc:205] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.454952 8844 raft_consensus.cc:1203] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.455049 9452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8384cbf3f368424d9f8a9060efe9c0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.456923 9452 tablet_copy_client.cc:783] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.457062 9452 tablet_copy_client.cc:647] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.459188 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:11.463033 8977 raft_consensus.cc:1203] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.466634 9610 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.469406 9611 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.473881 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=6ce0d065dc334ed5a9c8b1577ef35b3a] alter complete (version 3)
I20240503 07:59:11.475852 9613 tablet.cc:1722] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:11.478236 9452 tablet_copy_client.cc:523] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.479336 9452 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.483028 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:11.489073 9742 ts_tablet_manager.cc:902] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:11.489765 9742 tablet_copy_client.cc:308] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:11.495141 6319 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 7f417c61594348428ee9514ded49ee88 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:32897)
I20240503 07:59:11.495268 6319 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 7f417c61594348428ee9514ded49ee88 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:32897: session id = 9553fe3c4ccd4e91b7c176aad82fec09-7f417c61594348428ee9514ded49ee88
I20240503 07:59:11.496608 6319 tablet_copy_source_session.cc:205] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.498190 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f417c61594348428ee9514ded49ee88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.499709 9742 tablet_copy_client.cc:783] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.499908 9742 tablet_copy_client.cc:647] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.501295 9742 tablet_copy_client.cc:523] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.502326 9742 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.507448 8977 raft_consensus.cc:1203] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.511396 9485 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.511637 9485 tablet_bootstrap.cc:492] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:11.511816 9485 ts_tablet_manager.cc:1367] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.068s user 0.019s sys 0.001s
I20240503 07:59:11.512244 9485 raft_consensus.cc:348] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.512365 9485 raft_consensus.cc:729] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:11.512476 9485 consensus_queue.cc:260] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.512741 9485 ts_tablet_manager.cc:1398] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.513285 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-09c32a1e047844a3b09f2c790d735c9b received from {username='slave'} at 127.0.99.133:56365
I20240503 07:59:11.513360 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session c8d71787690245e681c85a087086e7ed-09c32a1e047844a3b09f2c790d735c9b on tablet 09c32a1e047844a3b09f2c790d735c9b with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.513779 9538 ts_tablet_manager.cc:902] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.514241 9538 tablet_copy_client.cc:308] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:11.514652 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 5631f3e97fe04013a2a7140f44260003 from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:53021)
I20240503 07:59:11.514734 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 5631f3e97fe04013a2a7140f44260003 from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:53021: session id = 3d08a06d53b14381bee6223c4e1a6027-5631f3e97fe04013a2a7140f44260003
I20240503 07:59:11.515527 6450 tablet_copy_source_session.cc:205] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.516345 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5631f3e97fe04013a2a7140f44260003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.518265 9538 tablet_copy_client.cc:783] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.518396 9538 tablet_copy_client.cc:647] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.519495 9538 tablet_copy_client.cc:523] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.520498 9538 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.528896 8712 raft_consensus.cc:1203] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.529271 9486 ts_tablet_manager.cc:902] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:11.529690 9486 tablet_copy_client.cc:308] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:11.530957 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 82de209205b342cfaac10506fdac9303 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:11.531059 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 82de209205b342cfaac10506fdac9303 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-82de209205b342cfaac10506fdac9303
I20240503 07:59:11.531937 6712 tablet_copy_source_session.cc:205] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.533142 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82de209205b342cfaac10506fdac9303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.535079 9486 tablet_copy_client.cc:783] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.535202 9486 tablet_copy_client.cc:647] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.535545 9452 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.535858 9452 tablet_bootstrap.cc:492] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:11.536048 9452 ts_tablet_manager.cc:1367] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.057s user 0.030s sys 0.000s
I20240503 07:59:11.536479 9452 raft_consensus.cc:348] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.536648 9452 raft_consensus.cc:729] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:11.536841 9452 consensus_queue.cc:260] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.537173 9452 ts_tablet_manager.cc:1398] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.537637 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-8384cbf3f368424d9f8a9060efe9c0f2 received from {username='slave'} at 127.0.99.134:40145
I20240503 07:59:11.537711 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-8384cbf3f368424d9f8a9060efe9c0f2 on tablet 8384cbf3f368424d9f8a9060efe9c0f2 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.540290 9742 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.540577 9742 tablet_bootstrap.cc:492] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:11.540760 9742 ts_tablet_manager.cc:1367] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.038s user 0.031s sys 0.000s
I20240503 07:59:11.541276 9511 ts_tablet_manager.cc:902] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:11.541715 9511 tablet_copy_client.cc:308] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:11.542104 9485 ts_tablet_manager.cc:902] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:11.542474 9485 tablet_copy_client.cc:308] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:11.542863 9486 tablet_copy_client.cc:523] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.542886 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 39b01318dc1543cba2cc991e9db91c28 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:54869)
I20240503 07:59:11.542971 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 39b01318dc1543cba2cc991e9db91c28 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:54869: session id = c8d71787690245e681c85a087086e7ed-39b01318dc1543cba2cc991e9db91c28
I20240503 07:59:11.543764 6712 tablet_copy_source_session.cc:205] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.544025 9486 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.541165 9742 raft_consensus.cc:348] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.545013 9742 raft_consensus.cc:729] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:11.545142 9742 consensus_queue.cc:260] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.545361 9538 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.545423 9742 ts_tablet_manager.cc:1398] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:11.545572 9538 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:11.545696 9538 ts_tablet_manager.cc:1367] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 07:59:11.545964 6319 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-7f417c61594348428ee9514ded49ee88 received from {username='slave'} at 127.0.99.134:32897
I20240503 07:59:11.546047 6319 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-7f417c61594348428ee9514ded49ee88 on tablet 7f417c61594348428ee9514ded49ee88 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.546137 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 2f943ecfe59f432c861a54b16c62572f from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:43649)
I20240503 07:59:11.546093 9538 raft_consensus.cc:348] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.546200 9538 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:11.546334 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 2f943ecfe59f432c861a54b16c62572f from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:43649: session id = 3d08a06d53b14381bee6223c4e1a6027-2f943ecfe59f432c861a54b16c62572f
I20240503 07:59:11.546290 9538 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.546674 9538 ts_tablet_manager.cc:1398] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.547135 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-5631f3e97fe04013a2a7140f44260003 received from {username='slave'} at 127.0.99.136:53021
I20240503 07:59:11.547209 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-5631f3e97fe04013a2a7140f44260003 on tablet 5631f3e97fe04013a2a7140f44260003 with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:11.547199 6318 tablet_copy_source_session.cc:205] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.548254 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f943ecfe59f432c861a54b16c62572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.548360 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b01318dc1543cba2cc991e9db91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.550139 9511 tablet_copy_client.cc:783] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.550221 9485 tablet_copy_client.cc:783] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.550264 9511 tablet_copy_client.cc:647] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.550328 9485 tablet_copy_client.cc:647] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.553999 9511 tablet_copy_client.cc:523] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.555050 9511 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.555202 9485 tablet_copy_client.cc:523] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.556149 9485 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.557902 9742 ts_tablet_manager.cc:902] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:11.558255 9742 tablet_copy_client.cc:308] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:11.558665 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet fd67fb07203e4e99919ec0317bdf077d from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:38577)
I20240503 07:59:11.558753 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet fd67fb07203e4e99919ec0317bdf077d from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:38577: session id = 9553fe3c4ccd4e91b7c176aad82fec09-fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.559545 6712 tablet_copy_source_session.cc:205] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.560292 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd67fb07203e4e99919ec0317bdf077d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.562218 9742 tablet_copy_client.cc:783] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.562338 9742 tablet_copy_client.cc:647] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.568670 9742 tablet_copy_client.cc:523] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.569813 9742 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.576198 9576 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.582064 9485 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.582294 9485 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:11.582432 9485 ts_tablet_manager.cc:1367] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.003s
I20240503 07:59:11.582640 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:11.582762 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:11.582826 9485 raft_consensus.cc:348] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.582947 9485 raft_consensus.cc:729] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:11.583042 9485 consensus_queue.cc:260] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:11.583298 9485 ts_tablet_manager.cc:1398] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.584707 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-39b01318dc1543cba2cc991e9db91c28 received from {username='slave'} at 127.0.99.133:54869
I20240503 07:59:11.584805 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session c8d71787690245e681c85a087086e7ed-39b01318dc1543cba2cc991e9db91c28 on tablet 39b01318dc1543cba2cc991e9db91c28 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:11.585892 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 442bf6b34d6b48c29fc497372128eaf0
I20240503 07:59:11.586176 9576 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.586493 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:11.586642 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.586772 9398 tablet_replica.cc:330] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:11.586877 9398 raft_consensus.cc:2229] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.586843 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 442bf6b34d6b48c29fc497372128eaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:11.587123 9398 raft_consensus.cc:2258] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.587164 9175 tablet_replica.cc:330] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.587294 9175 raft_consensus.cc:2229] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.587399 9175 raft_consensus.cc:2258] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.587679 9398 ts_tablet_manager.cc:1875] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.588191 9175 ts_tablet_manager.cc:1875] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.589711 9398 ts_tablet_manager.cc:1888] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.589776 9398 log.cc:1196] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/442bf6b34d6b48c29fc497372128eaf0
I20240503 07:59:11.589991 9398 ts_tablet_manager.cc:1909] T 442bf6b34d6b48c29fc497372128eaf0 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:11.590111 9486 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.590188 9175 ts_tablet_manager.cc:1888] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.590240 9175 log.cc:1196] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/442bf6b34d6b48c29fc497372128eaf0
I20240503 07:59:11.590332 9486 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:11.590477 9486 ts_tablet_manager.cc:1367] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.007s
I20240503 07:59:11.590519 9175 ts_tablet_manager.cc:1909] T 442bf6b34d6b48c29fc497372128eaf0 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.591142 9486 raft_consensus.cc:348] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.591318 9486 raft_consensus.cc:729] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:11.591477 9486 consensus_queue.cc:260] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.591938 9486 ts_tablet_manager.cc:1398] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.592005 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table_new_name [id=6ce0d065dc334ed5a9c8b1577ef35b3a]) successfully deleted
I20240503 07:59:11.593204 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-82de209205b342cfaac10506fdac9303 received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:11.593264 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-82de209205b342cfaac10506fdac9303 on tablet 82de209205b342cfaac10506fdac9303 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:11.593339 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table_new_name [id=6ce0d065dc334ed5a9c8b1577ef35b3a]) successfully deleted
I20240503 07:59:11.593381 9782 ts_tablet_manager.cc:902] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:11.594149 9782 tablet_copy_client.cc:308] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:11.587708 9780 tablet_replica.cc:330] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:11.594870 9780 raft_consensus.cc:2229] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.595017 9780 raft_consensus.cc:2258] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.595908 9780 ts_tablet_manager.cc:1875] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.598402 9780 ts_tablet_manager.cc:1888] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.598479 9780 log.cc:1196] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/442bf6b34d6b48c29fc497372128eaf0
I20240503 07:59:11.598724 9780 ts_tablet_manager.cc:1909] T 442bf6b34d6b48c29fc497372128eaf0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:11.599565 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 442bf6b34d6b48c29fc497372128eaf0 (table rebalancer_dynamic_table_new_name [id=6ce0d065dc334ed5a9c8b1577ef35b3a]) successfully deleted
I20240503 07:59:11.592449 9511 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.602166 9511 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:11.602312 9511 ts_tablet_manager.cc:1367] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.047s user 0.023s sys 0.000s
I20240503 07:59:11.602672 9511 raft_consensus.cc:348] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.602759 9511 raft_consensus.cc:729] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:11.602844 9511 consensus_queue.cc:260] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:11.603070 9511 ts_tablet_manager.cc:1398] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.604508 9742 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.604743 9742 tablet_bootstrap.cc:492] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:11.604749 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-2f943ecfe59f432c861a54b16c62572f received from {username='slave'} at 127.0.99.136:43649
I20240503 07:59:11.604853 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-2f943ecfe59f432c861a54b16c62572f on tablet 2f943ecfe59f432c861a54b16c62572f with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:11.604928 9742 ts_tablet_manager.cc:1367] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.004s
I20240503 07:59:11.605630 9742 raft_consensus.cc:348] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.605728 9742 raft_consensus.cc:729] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:11.605818 9742 consensus_queue.cc:260] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.606057 9742 ts_tablet_manager.cc:1398] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:11.606447 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-fd67fb07203e4e99919ec0317bdf077d received from {username='slave'} at 127.0.99.134:38577
I20240503 07:59:11.606508 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-fd67fb07203e4e99919ec0317bdf077d on tablet fd67fb07203e4e99919ec0317bdf077d with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.606801 6560 consensus_queue.cc:237] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:11.608177 6690 raft_consensus.cc:1261] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.608364 6296 raft_consensus.cc:1261] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:11.608582 9278 consensus_queue.cc:1035] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.608723 8893 consensus_queue.cc:1035] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:11.609844 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 5d95cc5335534f02aa62b00fb8b9c33c from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:33533)
I20240503 07:59:11.609923 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 5d95cc5335534f02aa62b00fb8b9c33c from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:33533: session id = 616fdda6ba714cb7be01a564de2b9db2-5d95cc5335534f02aa62b00fb8b9c33c
I20240503 07:59:11.610318 8575 raft_consensus.cc:1203] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.610778 6581 tablet_copy_source_session.cc:205] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.611536 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d95cc5335534f02aa62b00fb8b9c33c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:11.611747 6497 consensus_peers.cc:485] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f05300c9002408084f0b47d5e47591d. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:11.613457 9782 tablet_copy_client.cc:783] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.613592 9782 tablet_copy_client.cc:647] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.615697 9782 tablet_copy_client.cc:523] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.616732 9782 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.620221 9277 raft_consensus.cc:2932] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.620916 6296 raft_consensus.cc:2932] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.623343 6156 catalog_manager.cc:5484] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.624899 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 07:59:11.624992 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 07:59:11.627581 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.627873 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet f8f08a2db2da42a8b0d72ec82daa8346
I20240503 07:59:11.628024 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet 5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.628135 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet fd67fb07203e4e99919ec0317bdf077d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:11.628345 9398 tablet_replica.cc:330] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:11.628449 9398 raft_consensus.cc:2229] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:11.628525 9398 raft_consensus.cc:2258] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:11.628574 6671 tablet_service.cc:1514] Processing DeleteTablet for tablet fd67fb07203e4e99919ec0317bdf077d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.628643 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet fd67fb07203e4e99919ec0317bdf077d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:11.628695 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.628824 9175 tablet_replica.cc:330] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.628914 9178 tablet_replica.cc:330] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:11.628945 9175 raft_consensus.cc:2229] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.629053 9178 raft_consensus.cc:2229] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.629078 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5710dff79184444797bb46d981ae6f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:11.629163 9175 raft_consensus.cc:2258] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.629189 9178 raft_consensus.cc:2258] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.629281 9398 ts_tablet_manager.cc:1875] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.628839 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 5710dff79184444797bb46d981ae6f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.629683 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5710dff79184444797bb46d981ae6f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:11.630007 9178 ts_tablet_manager.cc:1875] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.631017 9398 ts_tablet_manager.cc:1888] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.631083 9398 log.cc:1196] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.631311 9398 ts_tablet_manager.cc:1909] T fd67fb07203e4e99919ec0317bdf077d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:11.631815 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 5710dff79184444797bb46d981ae6f55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.631925 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet fd67fb07203e4e99919ec0317bdf077d (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.632059 9178 ts_tablet_manager.cc:1888] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.632128 9178 log.cc:1196] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.632347 9178 ts_tablet_manager.cc:1909] T fd67fb07203e4e99919ec0317bdf077d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:11.632506 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet fd67fb07203e4e99919ec0317bdf077d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.632622 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:11.632730 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f08a2db2da42a8b0d72ec82daa8346 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:11 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:11.632881 9178 tablet_replica.cc:330] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:11.632946 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet fd67fb07203e4e99919ec0317bdf077d (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.632990 9178 raft_consensus.cc:2229] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.633018 9177 tablet_replica.cc:330] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.633098 9178 raft_consensus.cc:2258] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.633137 9177 raft_consensus.cc:2229] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.633267 9177 raft_consensus.cc:2258] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.633428 9176 tablet_replica.cc:330] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:11.633546 9176 raft_consensus.cc:2229] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.633658 9176 raft_consensus.cc:2258] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.634095 9176 ts_tablet_manager.cc:1875] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.635969 9176 ts_tablet_manager.cc:1888] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.636027 9176 log.cc:1196] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8f08a2db2da42a8b0d72ec82daa8346
I20240503 07:59:11.636241 9176 ts_tablet_manager.cc:1909] T f8f08a2db2da42a8b0d72ec82daa8346 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:11.636831 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet f8f08a2db2da42a8b0d72ec82daa8346 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.637430 9177 ts_tablet_manager.cc:1875] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.638020 6690 raft_consensus.cc:2932] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:11.639405 9177 ts_tablet_manager.cc:1888] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.639467 9177 log.cc:1196] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.639689 9177 ts_tablet_manager.cc:1909] T 5710dff79184444797bb46d981ae6f55 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.640199 9177 tablet_replica.cc:330] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.640303 9177 raft_consensus.cc:2229] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.640410 9177 raft_consensus.cc:2258] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.640981 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 5710dff79184444797bb46d981ae6f55 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.641543 9175 ts_tablet_manager.cc:1875] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.643422 9175 ts_tablet_manager.cc:1888] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.643477 9175 log.cc:1196] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.643687 9175 ts_tablet_manager.cc:1909] T fd67fb07203e4e99919ec0317bdf077d P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.644212 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet fd67fb07203e4e99919ec0317bdf077d (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.644373 9175 tablet_replica.cc:330] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.644610 9175 raft_consensus.cc:2229] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.644937 9175 raft_consensus.cc:2258] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.645717 9175 ts_tablet_manager.cc:1875] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.646136 9177 ts_tablet_manager.cc:1875] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.646608 9178 ts_tablet_manager.cc:1875] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.648315 9177 ts_tablet_manager.cc:1888] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.650074 9175 ts_tablet_manager.cc:1888] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.650188 9177 log.cc:1196] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd67fb07203e4e99919ec0317bdf077d
I20240503 07:59:11.650466 9177 ts_tablet_manager.cc:1909] T fd67fb07203e4e99919ec0317bdf077d P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.650554 9175 log.cc:1196] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8f08a2db2da42a8b0d72ec82daa8346
I20240503 07:59:11.650815 9175 ts_tablet_manager.cc:1909] T f8f08a2db2da42a8b0d72ec82daa8346 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.651324 9177 tablet_replica.cc:330] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:11.651392 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet fd67fb07203e4e99919ec0317bdf077d (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.651669 9178 ts_tablet_manager.cc:1888] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.651772 9178 log.cc:1196] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.651765 9782 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.651978 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet f8f08a2db2da42a8b0d72ec82daa8346 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.651993 9782 tablet_bootstrap.cc:492] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:11.652143 9178 ts_tablet_manager.cc:1909] T 5710dff79184444797bb46d981ae6f55 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:11.652168 9175 tablet_replica.cc:330] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.652160 9782 ts_tablet_manager.cc:1367] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.009s
I20240503 07:59:11.652335 9177 raft_consensus.cc:2229] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.652485 9175 raft_consensus.cc:2229] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:11.652578 9782 raft_consensus.cc:348] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.652634 9177 raft_consensus.cc:2258] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.652714 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 5710dff79184444797bb46d981ae6f55 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.652740 9782 raft_consensus.cc:729] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:11.652941 9782 consensus_queue.cc:260] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.653122 9175 raft_consensus.cc:2258] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.653296 9782 ts_tablet_manager.cc:1398] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:11.653558 9177 ts_tablet_manager.cc:1875] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.653942 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-5d95cc5335534f02aa62b00fb8b9c33c received from {username='slave'} at 127.0.99.135:33533
I20240503 07:59:11.654004 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-5d95cc5335534f02aa62b00fb8b9c33c on tablet 5d95cc5335534f02aa62b00fb8b9c33c with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:11.654230 9796 tablet_replica.cc:330] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:11.654347 9796 raft_consensus.cc:2229] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:11.654460 9796 raft_consensus.cc:2258] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:11.655139 9796 ts_tablet_manager.cc:1875] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.656433 9177 ts_tablet_manager.cc:1888] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:11.656497 9177 log.cc:1196] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f8f08a2db2da42a8b0d72ec82daa8346
I20240503 07:59:11.656716 9177 ts_tablet_manager.cc:1909] T f8f08a2db2da42a8b0d72ec82daa8346 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:11.657049 9175 ts_tablet_manager.cc:1875] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:11.657176 9796 ts_tablet_manager.cc:1888] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.657230 9796 log.cc:1196] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.657258 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet f8f08a2db2da42a8b0d72ec82daa8346 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.657481 9796 ts_tablet_manager.cc:1909] T 5710dff79184444797bb46d981ae6f55 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:11.658746 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 5710dff79184444797bb46d981ae6f55 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.658960 9175 ts_tablet_manager.cc:1888] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:11.659031 9175 log.cc:1196] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5710dff79184444797bb46d981ae6f55
I20240503 07:59:11.659281 9175 ts_tablet_manager.cc:1909] T 5710dff79184444797bb46d981ae6f55 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:11.659873 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 5710dff79184444797bb46d981ae6f55 (table rebalance_test_table_6 [id=7441f4d1d4d2427288686c47216a9c95]) successfully deleted
I20240503 07:59:11.680354 9782 ts_tablet_manager.cc:902] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:11.680732 9782 tablet_copy_client.cc:308] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:11.681201 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet d4308bc87be24f28b6566ce76cae6e50 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:34853)
I20240503 07:59:11.681289 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet d4308bc87be24f28b6566ce76cae6e50 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:34853: session id = 616fdda6ba714cb7be01a564de2b9db2-d4308bc87be24f28b6566ce76cae6e50
I20240503 07:59:11.682082 6318 tablet_copy_source_session.cc:205] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:11.682827 9782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4308bc87be24f28b6566ce76cae6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.684808 9782 tablet_copy_client.cc:783] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:11.684954 9782 tablet_copy_client.cc:647] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:11.686105 9782 tablet_copy_client.cc:523] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:11.687047 9782 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.688051 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:11.693876 8560 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c954d610f423594cf9d4575f38573 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]), partition=
I20240503 07:59:11.694096 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c954d610f423594cf9d4575f38573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.694110 8824 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c954d610f423594cf9d4575f38573 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]), partition=
I20240503 07:59:11.694111 8691 tablet_service.cc:1467] Processing CreateTablet for tablet aa4c954d610f423594cf9d4575f38573 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]), partition=
I20240503 07:59:11.694296 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c954d610f423594cf9d4575f38573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.694296 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa4c954d610f423594cf9d4575f38573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.696672 9840 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.696902 9839 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.697731 9840 tablet_bootstrap.cc:654] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.697731 9839 tablet_bootstrap.cc:654] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.698869 9840 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.698951 9840 ts_tablet_manager.cc:1367] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.699398 9840 raft_consensus.cc:348] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.699487 9840 raft_consensus.cc:374] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.699515 9840 raft_consensus.cc:729] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.699640 9840 consensus_queue.cc:260] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.700723 9840 ts_tablet_manager.cc:1398] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.700982 9838 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:11.703413 9838 tablet_bootstrap.cc:654] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.704352 9838 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:11.704566 9838 ts_tablet_manager.cc:1367] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:11.705471 9838 raft_consensus.cc:348] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.705564 9838 raft_consensus.cc:374] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.705592 9838 raft_consensus.cc:729] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.705724 9838 consensus_queue.cc:260] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.706687 9838 ts_tablet_manager.cc:1398] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.711266 9782 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:11.716974 9782 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:11.717072 9839 tablet_bootstrap.cc:492] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:11.717245 9839 ts_tablet_manager.cc:1367] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:11.717275 9782 ts_tablet_manager.cc:1367] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.001s
I20240503 07:59:11.717658 9839 raft_consensus.cc:348] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.717754 9782 raft_consensus.cc:348] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.717861 9839 raft_consensus.cc:374] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.717865 9782 raft_consensus.cc:729] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:11.717949 9839 raft_consensus.cc:729] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.718011 9782 consensus_queue.cc:260] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:11.718065 9839 consensus_queue.cc:260] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:11.718292 9782 ts_tablet_manager.cc:1398] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.718380 9839 ts_tablet_manager.cc:1398] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.718716 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-d4308bc87be24f28b6566ce76cae6e50 received from {username='slave'} at 127.0.99.135:34853
I20240503 07:59:11.718806 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-d4308bc87be24f28b6566ce76cae6e50 on tablet d4308bc87be24f28b6566ce76cae6e50 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:11.740093 8844 raft_consensus.cc:1203] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.779771 8575 raft_consensus.cc:1203] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.792330 9803 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.795832 8575 raft_consensus.cc:3037] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.798157 9464 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.802654 9248 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.805651 9189 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:11.817962 8977 raft_consensus.cc:1203] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.838447 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:11.838552 8575 raft_consensus.cc:1203] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.845290 6278 tablet_service.cc:1467] Processing CreateTablet for tablet f635e30fb6974fa1a79d8a46eecc3d5e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.845566 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f635e30fb6974fa1a79d8a46eecc3d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.845624 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 9a70db6c0679432dbef5e65f5ea55423 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.845849 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a70db6c0679432dbef5e65f5ea55423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.845796 8691 tablet_service.cc:1467] Processing CreateTablet for tablet f635e30fb6974fa1a79d8a46eecc3d5e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.846029 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f635e30fb6974fa1a79d8a46eecc3d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.847889 9840 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.848045 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 09866b93f75545a19c9451556254da48 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.848249 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09866b93f75545a19c9451556254da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.848698 9840 tablet_bootstrap.cc:654] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.850634 9840 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.850723 9840 ts_tablet_manager.cc:1367] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:11.851363 9840 raft_consensus.cc:348] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.851434 9840 raft_consensus.cc:374] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.851454 9840 raft_consensus.cc:729] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.851538 9840 consensus_queue.cc:260] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.851749 9840 ts_tablet_manager.cc:1398] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:11.851888 9840 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:11.852632 9840 tablet_bootstrap.cc:654] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.854483 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 9a70db6c0679432dbef5e65f5ea55423 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.854538 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 9a70db6c0679432dbef5e65f5ea55423 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.854727 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a70db6c0679432dbef5e65f5ea55423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.854760 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a70db6c0679432dbef5e65f5ea55423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.856673 9839 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.857511 9839 tablet_bootstrap.cc:654] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.857782 9937 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:11.858716 9839 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:11.858800 9839 ts_tablet_manager.cc:1367] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:11.858825 9937 tablet_bootstrap.cc:654] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.859223 9839 raft_consensus.cc:348] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.859342 9839 raft_consensus.cc:374] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.859382 9839 raft_consensus.cc:729] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.859498 9839 consensus_queue.cc:260] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.859757 9839 ts_tablet_manager.cc:1398] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.859997 9937 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:11.860118 9937 ts_tablet_manager.cc:1367] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:11.860522 9937 raft_consensus.cc:348] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.860532 8956 tablet_service.cc:1467] Processing CreateTablet for tablet f635e30fb6974fa1a79d8a46eecc3d5e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.860679 9937 raft_consensus.cc:374] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.860646 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 09866b93f75545a19c9451556254da48 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.860740 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f635e30fb6974fa1a79d8a46eecc3d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.860759 9937 raft_consensus.cc:729] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.860882 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09866b93f75545a19c9451556254da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.860901 9937 consensus_queue.cc:260] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.861202 9937 ts_tablet_manager.cc:1398] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:11.862483 9839 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:11.862722 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 09866b93f75545a19c9451556254da48 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:11.862982 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09866b93f75545a19c9451556254da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:11.863428 9839 tablet_bootstrap.cc:654] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.864280 9839 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:11.864347 9839 ts_tablet_manager.cc:1367] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:11.864717 9839 raft_consensus.cc:348] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.864843 9839 raft_consensus.cc:374] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.864879 9839 raft_consensus.cc:729] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.864984 9839 consensus_queue.cc:260] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.865243 9839 ts_tablet_manager.cc:1398] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.865278 9938 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:11.865806 9941 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.866907 9945 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:11.867731 9945 tablet_bootstrap.cc:654] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.869096 9938 tablet_bootstrap.cc:654] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.869367 9945 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:11.869460 9945 ts_tablet_manager.cc:1367] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:11.869772 9941 tablet_bootstrap.cc:654] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.869845 9945 raft_consensus.cc:348] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.869946 9945 raft_consensus.cc:374] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.869980 9945 raft_consensus.cc:729] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.870035 9938 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:11.870113 9938 ts_tablet_manager.cc:1367] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:11.870082 9945 consensus_queue.cc:260] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.870314 9945 ts_tablet_manager.cc:1398] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.870483 9938 raft_consensus.cc:348] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.870560 9938 raft_consensus.cc:374] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.870579 9938 raft_consensus.cc:729] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.870671 9938 consensus_queue.cc:260] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.870792 9840 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:11.870887 9840 ts_tablet_manager.cc:1367] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:11.870911 9938 ts_tablet_manager.cc:1398] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.871351 9840 raft_consensus.cc:348] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.871444 9840 raft_consensus.cc:374] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.871472 9840 raft_consensus.cc:729] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.871585 9840 consensus_queue.cc:260] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.871840 9840 ts_tablet_manager.cc:1398] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.871872 9941 tablet_bootstrap.cc:492] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:11.871956 9941 ts_tablet_manager.cc:1367] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:11.874145 9941 raft_consensus.cc:348] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.874377 9941 raft_consensus.cc:374] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.874639 9941 raft_consensus.cc:729] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.874804 9941 consensus_queue.cc:260] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.875125 9941 ts_tablet_manager.cc:1398] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:11.875326 9941 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:11.876271 9941 tablet_bootstrap.cc:654] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:11.889067 9189 raft_consensus.cc:482] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.889161 9189 raft_consensus.cc:504] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.889508 9189 leader_election.cc:288] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:11.889814 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:11.889861 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:11.889937 6690 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:11.889971 8844 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:11.890182 8647 leader_election.cc:302] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:11.890374 9189 raft_consensus.cc:2781] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.890430 9189 raft_consensus.cc:482] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.890461 9189 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.891309 9189 raft_consensus.cc:504] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.891676 9189 leader_election.cc:288] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:11.891956 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:11.892072 6690 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.892047 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:11.892159 8844 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.893041 8844 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.893041 6690 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:11.893381 8647 leader_election.cc:302] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:11.893561 9189 raft_consensus.cc:2781] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.893646 9189 raft_consensus.cc:686] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:11.893767 9189 consensus_queue.cc:237] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.895218 6151 catalog_manager.cc:5484] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:11.897380 9941 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:11.897455 9941 ts_tablet_manager.cc:1367] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240503 07:59:11.897812 9941 raft_consensus.cc:348] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.897922 9941 raft_consensus.cc:374] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:11.897944 9941 raft_consensus.cc:729] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:11.898052 9941 consensus_queue.cc:260] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:11.898285 9941 ts_tablet_manager.cc:1398] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:11.903929 8712 raft_consensus.cc:1203] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.904927 8895 raft_consensus.cc:1050] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:11.905419 8895 consensus_queue.cc:237] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:11.906666 8844 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.906785 6296 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.906841 6560 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.907043 9309 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.907294 8895 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.907569 9288 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.909595 9952 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.909999 6560 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.910118 8844 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.911691 6296 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.912758 6151 catalog_manager.cc:5484] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.917906 6428 consensus_queue.cc:237] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:11.918859 8844 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:11.918915 6296 raft_consensus.cc:1261] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:11.919252 9952 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:11.919416 8895 consensus_queue.cc:1035] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:11.920738 9288 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.921172 8844 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.922820 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 610160f6ba94483a9bdcf966c413e5d1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:11.922910 6154 catalog_manager.cc:5484] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:11.923266 6296 raft_consensus.cc:2932] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:11.925164 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:11.925665 9176 tablet_replica.cc:330] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:11.925812 9176 raft_consensus.cc:2229] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.925994 9176 raft_consensus.cc:2258] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.926800 9176 ts_tablet_manager.cc:1875] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:11.927198 9855 raft_consensus.cc:482] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:11.927281 9855 raft_consensus.cc:504] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.927702 9855 leader_election.cc:288] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.928002 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a70db6c0679432dbef5e65f5ea55423" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:11.928020 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a70db6c0679432dbef5e65f5ea55423" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:11.928113 6428 raft_consensus.cc:2454] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:11.928138 8977 raft_consensus.cc:2454] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:11.928375 8779 leader_election.cc:302] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:11.928522 9855 raft_consensus.cc:2781] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:11.928575 9855 raft_consensus.cc:482] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:11.928611 9855 raft_consensus.cc:3037] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.928862 9176 ts_tablet_manager.cc:1888] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:11.928951 9176 log.cc:1196] T 610160f6ba94483a9bdcf966c413e5d1 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:11.929378 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 610160f6ba94483a9bdcf966c413e5d1 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:11.929414 9855 raft_consensus.cc:504] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.929692 9855 leader_election.cc:288] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:11.929946 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a70db6c0679432dbef5e65f5ea55423" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:11.929988 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a70db6c0679432dbef5e65f5ea55423" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:11.930034 8977 raft_consensus.cc:3037] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.930070 6428 raft_consensus.cc:3037] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:11.930701 8977 raft_consensus.cc:2454] T 9a70db6c0679432dbef5e65f5ea55423 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:11.930714 6428 raft_consensus.cc:2454] T 9a70db6c0679432dbef5e65f5ea55423 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:11.930941 8778 leader_election.cc:302] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:11.931116 9855 raft_consensus.cc:2781] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:11.931207 9855 raft_consensus.cc:686] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:11.931341 9855 consensus_queue.cc:237] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:11.932690 6152 catalog_manager.cc:5484] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:11.951737 8977 raft_consensus.cc:1203] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:11.956727 9952 raft_consensus.cc:1050] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:11.957072 9952 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:11.958352 6560 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.958413 8977 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.958413 6690 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:11.958725 9288 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.958879 8895 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.959071 9309 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:11.960994 9288 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.961424 8977 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.962183 6690 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.963569 6560 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.963873 6154 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.966697 6428 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:11.967904 6560 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:11.967940 8977 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:11.968314 9288 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:11.968648 8895 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:11.970073 8895 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.970559 8977 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.970818 6560 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:11.972280 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a10085c34be4d31aae9f56a2ec1ba23 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:11.973022 6152 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:11.975157 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a10085c34be4d31aae9f56a2ec1ba23 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:11.975363 9175 tablet_replica.cc:330] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:11.975520 9175 raft_consensus.cc:2229] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:11.975656 9175 raft_consensus.cc:2258] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:11.976437 9175 ts_tablet_manager.cc:1875] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:11.978303 9175 ts_tablet_manager.cc:1888] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:11.978384 9175 log.cc:1196] T 2a10085c34be4d31aae9f56a2ec1ba23 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a10085c34be4d31aae9f56a2ec1ba23
I20240503 07:59:11.978798 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 2a10085c34be4d31aae9f56a2ec1ba23 (table rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]) successfully deleted
I20240503 07:59:12.003823 9189 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.003907 9189 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:12.004374 9189 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:12.004664 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:12.004794 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:12.004842 8977 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:12.004902 6296 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:12.005113 8647 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:12.005264 9189 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.005295 9277 raft_consensus.cc:1050] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:12.005309 9189 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.005344 9189 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.005721 9277 consensus_queue.cc:237] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.006137 9189 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:12.006448 9189 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:12.007004 6428 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.007203 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:12.007236 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:12.007297 6298 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.007329 8976 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.008015 8976 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:12.008132 6298 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:12.008278 8647 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:12.008426 9189 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.008500 9189 raft_consensus.cc:686] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:12.008620 9189 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:12.008616 9963 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.008917 8976 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.009250 8893 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.009974 6690 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.010121 6154 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.010820 8893 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.012387 9963 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.012754 8976 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.013132 6690 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.014278 6428 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.015820 6154 catalog_manager.cc:5484] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.018632 6560 consensus_queue.cc:237] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.019488 6428 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.019539 8976 raft_consensus.cc:1261] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.019852 9278 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.019997 8893 consensus_queue.cc:1035] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.021436 8893 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.022204 8976 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.022774 6430 raft_consensus.cc:2932] T 5538a709d8d440cb8f006a163f259389 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.024621 6151 catalog_manager.cc:5484] T 5538a709d8d440cb8f006a163f259389 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.025182 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5538a709d8d440cb8f006a163f259389 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.026552 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 5538a709d8d440cb8f006a163f259389 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.026751 9175 tablet_replica.cc:330] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.026897 9175 raft_consensus.cc:2229] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.027032 9175 raft_consensus.cc:2258] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.027657 9175 ts_tablet_manager.cc:1875] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.029482 9175 ts_tablet_manager.cc:1888] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.029531 9175 log.cc:1196] T 5538a709d8d440cb8f006a163f259389 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5538a709d8d440cb8f006a163f259389
I20240503 07:59:12.029935 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 5538a709d8d440cb8f006a163f259389 (table rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]) successfully deleted
I20240503 07:59:12.048911 8575 raft_consensus.cc:1203] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.051301 8844 raft_consensus.cc:1203] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.063513 8976 raft_consensus.cc:1203] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.067124 9855 raft_consensus.cc:482] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.067209 9855 raft_consensus.cc:504] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:12.067554 9855 leader_election.cc:288] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:12.067855 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c954d610f423594cf9d4575f38573" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:12.067921 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c954d610f423594cf9d4575f38573" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:12.068003 8575 raft_consensus.cc:2454] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:12.068054 8712 raft_consensus.cc:2454] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:12.068285 8776 leader_election.cc:302] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:12.068521 9855 raft_consensus.cc:2781] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.068581 9855 raft_consensus.cc:482] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.068624 9855 raft_consensus.cc:3037] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.069427 9278 raft_consensus.cc:1050] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:12.069633 9855 raft_consensus.cc:504] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:12.069900 9278 consensus_queue.cc:237] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.070020 9855 leader_election.cc:288] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:12.070269 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c954d610f423594cf9d4575f38573" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:12.070390 8575 raft_consensus.cc:3037] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.071244 8712 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.071362 8575 raft_consensus.cc:2454] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:12.071379 6430 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.071545 6298 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.071604 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa4c954d610f423594cf9d4575f38573" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:12.071682 8712 raft_consensus.cc:3037] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.071884 9963 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.072036 9277 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.072227 9278 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.072366 8776 leader_election.cc:302] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:12.072580 8712 raft_consensus.cc:2454] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:12.072629 9855 raft_consensus.cc:2781] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.072696 9855 raft_consensus.cc:686] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:12.072841 9855 consensus_queue.cc:237] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:12.074559 8893 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.074952 6298 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.075239 6155 catalog_manager.cc:5484] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.077047 6430 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.077358 8712 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.078859 6152 catalog_manager.cc:5484] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.081094 8712 raft_consensus.cc:1203] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.082738 6560 consensus_queue.cc:237] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.085543 8844 raft_consensus.cc:1203] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.087320 6430 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.087824 9277 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.088910 8712 raft_consensus.cc:1261] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.089223 9277 consensus_queue.cc:1035] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.089545 9277 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.090934 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e9ce52b9ae543cca72fb92efafc3c4d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.091305 6430 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.093400 8712 raft_consensus.cc:2932] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.094071 6151 catalog_manager.cc:5484] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.096904 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d with delete_type TABLET_DATA_TOMBSTONED (TS c167a0a433d34b74affed13cbf96e302 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.097152 9177 tablet_replica.cc:330] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.097306 9177 raft_consensus.cc:2229] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.097450 9177 raft_consensus.cc:2258] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.098179 9177 ts_tablet_manager.cc:1875] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.100379 9177 ts_tablet_manager.cc:1888] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.100461 9177 log.cc:1196] T 0e9ce52b9ae543cca72fb92efafc3c4d P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:12.102569 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 0e9ce52b9ae543cca72fb92efafc3c4d (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:12.103950 8692 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:12.110332 6278 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:12.113535 6541 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:12.116835 6410 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:12.117744 8824 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:12.123013 6672 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:12.123554 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:12.133212 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 07:59:12.133335 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 07:59:12.135803 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 29cf7d1a310e44b0aee029422b88f0e7
I20240503 07:59:12.136103 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet 09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:12.136371 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 62e54fbe89f843168b4ecc21307a6739
I20240503 07:59:12.136473 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.136636 9177 tablet_replica.cc:330] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.136842 9177 raft_consensus.cc:2229] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.136930 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.136998 9177 raft_consensus.cc:2258] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.137104 9175 tablet_replica.cc:330] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.137207 9175 raft_consensus.cc:2229] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.137300 9175 raft_consensus.cc:2258] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.137496 9177 ts_tablet_manager.cc:1875] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.137789 9175 ts_tablet_manager.cc:1875] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.138499 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c32a1e047844a3b09f2c790d735c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.138689 9178 tablet_replica.cc:330] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.138815 9178 raft_consensus.cc:2229] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.139048 9178 raft_consensus.cc:2258] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.139581 9178 ts_tablet_manager.cc:1875] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.139616 9177 ts_tablet_manager.cc:1888] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.139674 9177 log.cc:1196] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/29cf7d1a310e44b0aee029422b88f0e7
I20240503 07:59:12.139745 9175 ts_tablet_manager.cc:1888] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.139806 9175 log.cc:1196] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/29cf7d1a310e44b0aee029422b88f0e7
I20240503 07:59:12.139942 9177 ts_tablet_manager.cc:1909] T 29cf7d1a310e44b0aee029422b88f0e7 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.140033 9175 ts_tablet_manager.cc:1909] T 29cf7d1a310e44b0aee029422b88f0e7 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:12.140657 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 29cf7d1a310e44b0aee029422b88f0e7 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.140944 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 29cf7d1a310e44b0aee029422b88f0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.141199 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 29cf7d1a310e44b0aee029422b88f0e7 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.141275 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e54fbe89f843168b4ecc21307a6739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.141427 9176 tablet_replica.cc:330] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.141463 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c32a1e047844a3b09f2c790d735c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.141539 9176 raft_consensus.cc:2229] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.141705 9176 raft_consensus.cc:2258] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.141734 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c32a1e047844a3b09f2c790d735c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.141855 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e54fbe89f843168b4ecc21307a6739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.141875 9177 tablet_replica.cc:330] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.142000 9177 raft_consensus.cc:2229] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.142122 9177 raft_consensus.cc:2258] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.142168 9176 ts_tablet_manager.cc:1875] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.142984 9178 ts_tablet_manager.cc:1888] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.143050 9178 log.cc:1196] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:12.143294 9178 ts_tablet_manager.cc:1909] T 09c32a1e047844a3b09f2c790d735c9b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:12.143805 9176 ts_tablet_manager.cc:1888] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.143851 9176 log.cc:1196] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/62e54fbe89f843168b4ecc21307a6739
I20240503 07:59:12.143846 9178 tablet_replica.cc:330] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.143958 9178 raft_consensus.cc:2229] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.144030 9176 ts_tablet_manager.cc:1909] T 62e54fbe89f843168b4ecc21307a6739 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:12.144115 9178 raft_consensus.cc:2258] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.144310 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 62e54fbe89f843168b4ecc21307a6739 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.144485 9175 tablet_replica.cc:330] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.144619 9178 ts_tablet_manager.cc:1875] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.144694 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c32a1e047844a3b09f2c790d735c9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:12.145282 9796 tablet_replica.cc:330] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:12.145469 9796 raft_consensus.cc:2229] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:12.145613 9796 raft_consensus.cc:2258] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:12.145627 9176 tablet_replica.cc:330] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.145735 9176 raft_consensus.cc:2229] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.145850 9176 raft_consensus.cc:2258] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.146152 9796 ts_tablet_manager.cc:1875] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.146457 9176 ts_tablet_manager.cc:1875] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.148166 9796 ts_tablet_manager.cc:1888] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.148232 9796 log.cc:1196] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:12.148260 9175 raft_consensus.cc:2229] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.148392 9175 raft_consensus.cc:2258] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.148486 9796 ts_tablet_manager.cc:1909] T 09c32a1e047844a3b09f2c790d735c9b P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:12.149014 9175 ts_tablet_manager.cc:1875] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.150038 9178 ts_tablet_manager.cc:1888] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.150105 9178 log.cc:1196] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/29cf7d1a310e44b0aee029422b88f0e7
I20240503 07:59:12.150321 9178 ts_tablet_manager.cc:1909] T 29cf7d1a310e44b0aee029422b88f0e7 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:12.150698 9175 ts_tablet_manager.cc:1888] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.150766 9175 log.cc:1196] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/62e54fbe89f843168b4ecc21307a6739
I20240503 07:59:12.150991 9175 ts_tablet_manager.cc:1909] T 62e54fbe89f843168b4ecc21307a6739 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:12.151437 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 09c32a1e047844a3b09f2c790d735c9b (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.151530 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 09c32a1e047844a3b09f2c790d735c9b (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.151579 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 29cf7d1a310e44b0aee029422b88f0e7 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.151723 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 62e54fbe89f843168b4ecc21307a6739 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.152724 9176 ts_tablet_manager.cc:1888] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.152838 9176 log.cc:1196] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:12.153085 9176 ts_tablet_manager.cc:1909] T 09c32a1e047844a3b09f2c790d735c9b P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:12.153692 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 09c32a1e047844a3b09f2c790d735c9b (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.155517 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 62e54fbe89f843168b4ecc21307a6739 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.155885 9177 ts_tablet_manager.cc:1875] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.157721 9177 ts_tablet_manager.cc:1888] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.157796 9177 log.cc:1196] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/09c32a1e047844a3b09f2c790d735c9b
I20240503 07:59:12.158031 9177 ts_tablet_manager.cc:1909] T 09c32a1e047844a3b09f2c790d735c9b P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.158555 9177 tablet_replica.cc:330] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.158668 9177 raft_consensus.cc:2229] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.158704 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 09c32a1e047844a3b09f2c790d735c9b (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.158753 9177 raft_consensus.cc:2258] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.159125 9177 ts_tablet_manager.cc:1875] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.160693 9177 ts_tablet_manager.cc:1888] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.160759 9177 log.cc:1196] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/62e54fbe89f843168b4ecc21307a6739
I20240503 07:59:12.161398 8956 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:12.165159 8893 raft_consensus.cc:1050] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:12.165571 8893 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.167208 6690 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.167560 6430 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.167666 9277 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.168066 9177 ts_tablet_manager.cc:1909] T 62e54fbe89f843168b4ecc21307a6739 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.168471 9277 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.169994 8575 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.170418 9277 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.175853 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 62e54fbe89f843168b4ecc21307a6739 (table rebalance_test_table_8 [id=4c4628ca9d9a4e16a0d6d2c1a88adccc]) successfully deleted
I20240503 07:59:12.178846 9278 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.180424 6690 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.182077 6155 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.183660 6430 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.188498 8575 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.194111 6560 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.195132 8575 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.195132 6690 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.195506 9278 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.195664 8893 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.197242 9277 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.197715 6690 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.199087 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c597bc1a19274f55b9d60558db35fcae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.199963 8844 raft_consensus.cc:1203] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.201565 6155 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.202431 8575 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.205041 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet c597bc1a19274f55b9d60558db35fcae with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.205255 9178 tablet_replica.cc:330] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.205382 9178 raft_consensus.cc:2229] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.205523 9178 raft_consensus.cc:2258] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.208141 9178 ts_tablet_manager.cc:1875] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.209955 9178 ts_tablet_manager.cc:1888] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.210029 9178 log.cc:1196] T c597bc1a19274f55b9d60558db35fcae P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c597bc1a19274f55b9d60558db35fcae
I20240503 07:59:12.210585 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet c597bc1a19274f55b9d60558db35fcae (table rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]) successfully deleted
I20240503 07:59:12.229732 6560 consensus_queue.cc:237] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.231122 6298 raft_consensus.cc:1261] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.231120 6430 raft_consensus.cc:1261] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.231480 6497 consensus_peers.cc:485] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e3f58a27f2a4a2593e8219b1d7bd894. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.231549 9963 consensus_queue.cc:1035] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.231755 8893 consensus_queue.cc:1035] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.232229 10000 ts_tablet_manager.cc:902] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:12.232663 10000 tablet_copy_client.cc:308] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:12.233357 9278 raft_consensus.cc:2932] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.234735 6430 raft_consensus.cc:2932] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.234843 6298 raft_consensus.cc:2932] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.237648 6151 catalog_manager.cc:5484] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.239444 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 5f05300c9002408084f0b47d5e47591d from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:43151)
I20240503 07:59:12.239537 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 5f05300c9002408084f0b47d5e47591d from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:43151: session id = 9553fe3c4ccd4e91b7c176aad82fec09-5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.240366 6581 tablet_copy_source_session.cc:205] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.242522 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f05300c9002408084f0b47d5e47591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.244514 10000 tablet_copy_client.cc:783] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.244675 10000 tablet_copy_client.cc:647] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.245802 10000 tablet_copy_client.cc:523] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.246975 10000 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:12.247908 6560 consensus_queue.cc:237] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:12.249277 6690 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.249454 6298 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.249617 9963 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.252707 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 07:59:12.252949 6494 consensus_peers.cc:485] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d855e5edf05471983dfdbd40f37560f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.253037 8893 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.253536 9963 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.256570 6690 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.258515 6155 catalog_manager.cc:5484] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.263198 6298 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.265780 8712 raft_consensus.cc:1261] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.266187 9855 consensus_queue.cc:1035] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.268239 8575 raft_consensus.cc:1261] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.268393 6154 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]). Expected version 1 got 0
I20240503 07:59:12.268643 9317 consensus_queue.cc:1035] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.269616 10023 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.269951 10023 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:12.271387 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]). Expected version 1 got 0
I20240503 07:59:12.272034 6155 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]). Expected version 1 got 0
I20240503 07:59:12.274732 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba] alter complete (version 1)
I20240503 07:59:12.274749 10025 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.274996 10025 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:12.276559 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba]). Expected version 1 got 0
I20240503 07:59:12.279362 10026 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.279644 10026 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:12.284124 10000 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.284550 10000 tablet_bootstrap.cc:492] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:12.284719 10000 ts_tablet_manager.cc:1367] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.038s user 0.019s sys 0.004s
I20240503 07:59:12.285236 10000 raft_consensus.cc:348] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.285352 10000 raft_consensus.cc:729] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:12.285476 10000 consensus_queue.cc:260] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.285802 10000 ts_tablet_manager.cc:1398] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.286437 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-5f05300c9002408084f0b47d5e47591d received from {username='slave'} at 127.0.99.134:43151
I20240503 07:59:12.286510 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-5f05300c9002408084f0b47d5e47591d on tablet 5f05300c9002408084f0b47d5e47591d with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:12.308192 9958 raft_consensus.cc:1050] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:12.308557 9958 consensus_queue.cc:237] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.309957 6690 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.310134 6560 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.310540 8895 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.310778 8712 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.310844 8895 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.311111 9288 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.313596 9288 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.315330 6690 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.316232 6154 catalog_manager.cc:5484] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.316394 9309 raft_consensus.cc:1050] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:12.316726 9309 consensus_queue.cc:237] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.317183 8712 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.318178 8575 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.318539 6298 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.318523 8895 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.319129 9952 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.319801 6560 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.321197 6561 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.321902 6430 consensus_queue.cc:237] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.322327 9288 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.323627 9347 raft_consensus.cc:1050] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:12.323791 6560 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.324079 9347 consensus_queue.cc:237] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.324132 8895 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.325302 6559 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.325428 6296 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.325879 8844 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.325878 9968 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.326182 9968 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.326323 8712 raft_consensus.cc:1261] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.326326 8077 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.327427 9952 consensus_queue.cc:1035] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.328581 9952 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.329962 8712 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.330099 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8384cbf3f368424d9f8a9060efe9c0f2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.331113 6559 raft_consensus.cc:2932] T 8384cbf3f368424d9f8a9060efe9c0f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.331125 8077 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.331770 6155 catalog_manager.cc:5484] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.333621 6296 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.334503 6151 catalog_manager.cc:5484] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.335177 8844 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.335325 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 8384cbf3f368424d9f8a9060efe9c0f2 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.335606 9175 tablet_replica.cc:330] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.335737 9175 raft_consensus.cc:2229] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.335880 9175 raft_consensus.cc:2258] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.337961 6690 consensus_queue.cc:237] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.338393 9175 ts_tablet_manager.cc:1875] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.339061 8844 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.339499 9968 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.339745 6559 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.340363 9175 ts_tablet_manager.cc:1888] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.340421 9175 log.cc:1196] T 8384cbf3f368424d9f8a9060efe9c0f2 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8384cbf3f368424d9f8a9060efe9c0f2
I20240503 07:59:12.341027 6298 raft_consensus.cc:1261] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.341467 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 8384cbf3f368424d9f8a9060efe9c0f2 (table rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]) successfully deleted
I20240503 07:59:12.341544 9347 consensus_queue.cc:1035] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.343427 9952 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.343802 9347 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.344717 8844 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.345856 8575 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.346549 6298 raft_consensus.cc:2932] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.347965 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 065d5d0851e840939802896f9bddb84c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.348448 6152 catalog_manager.cc:5484] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.349325 6298 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.351590 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 065d5d0851e840939802896f9bddb84c with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.352195 6155 catalog_manager.cc:5484] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.352280 9176 tablet_replica.cc:330] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.352397 9176 raft_consensus.cc:2229] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.352526 9176 raft_consensus.cc:2258] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.353865 9176 ts_tablet_manager.cc:1875] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.355726 9309 raft_consensus.cc:1050] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:12.355837 9176 ts_tablet_manager.cc:1888] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.355906 9176 log.cc:1196] T 065d5d0851e840939802896f9bddb84c P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/065d5d0851e840939802896f9bddb84c
I20240503 07:59:12.356052 9309 consensus_queue.cc:237] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.356346 6561 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.356877 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 065d5d0851e840939802896f9bddb84c (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:12.358984 8976 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.359088 6561 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.359292 6298 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.359439 8895 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.359783 9309 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.360080 9958 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.362080 8895 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.362417 6561 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.362449 6298 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.363879 8976 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.364706 6154 catalog_manager.cc:5484] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.367961 6430 consensus_queue.cc:237] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.368906 6430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47150
I20240503 07:59:12.369031 6430 raft_consensus.cc:595] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:12.369143 6298 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.369138 8976 raft_consensus.cc:1261] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.369431 9309 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.369556 9958 consensus_queue.cc:1035] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.371527 9958 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.371965 6298 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.374234 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 93eb2db48ef449a486a0347a76a8a5ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.374704 6156 catalog_manager.cc:5484] T 93eb2db48ef449a486a0347a76a8a5ea P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.375121 8976 raft_consensus.cc:2932] T 93eb2db48ef449a486a0347a76a8a5ea P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.376653 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 93eb2db48ef449a486a0347a76a8a5ea with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.376829 9176 tablet_replica.cc:330] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.376986 9176 raft_consensus.cc:2229] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.377132 9176 raft_consensus.cc:2258] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.378297 9855 consensus_queue.cc:1035] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.378748 9176 ts_tablet_manager.cc:1875] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.379756 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:12.380611 9176 ts_tablet_manager.cc:1888] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.380683 9176 log.cc:1196] T 93eb2db48ef449a486a0347a76a8a5ea P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/93eb2db48ef449a486a0347a76a8a5ea
I20240503 07:59:12.381251 10095 consensus_queue.cc:1035] T 9a70db6c0679432dbef5e65f5ea55423 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.382995 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 93eb2db48ef449a486a0347a76a8a5ea (table rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]) successfully deleted
I20240503 07:59:12.385416 10023 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.385692 10023 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:12.387282 10025 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.387549 10025 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:12.388321 10026 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:12.388608 10026 tablet.cc:1620] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:12.391002 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=f169430077e4414490b89649993f03ba] alter complete (version 2)
I20240503 07:59:12.440399 6430 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:12.441815 6561 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.441823 6690 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.442159 9309 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.442354 6365 consensus_peers.cc:485] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c01e3ce57874e24ae75a41243fdeaaf. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.442361 8895 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.444193 9309 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.444609 6561 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.444873 6690 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.447201 6154 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.453974 6428 consensus_queue.cc:237] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:12.455041 6561 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.455214 6365 consensus_peers.cc:485] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5604af852814da58385f072eb9600aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.455370 8895 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.455474 6298 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.455861 8895 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.456918 9288 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.457337 6561 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.459586 6151 catalog_manager.cc:5484] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.460695 6298 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.466719 6298 consensus_queue.cc:237] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.467814 6690 raft_consensus.cc:1261] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.468102 6428 raft_consensus.cc:1261] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.468160 9803 consensus_queue.cc:1035] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.468334 6235 consensus_peers.cc:485] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11fbfb84cef74316a77b3ccfab49a19b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.468521 9469 consensus_queue.cc:1035] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.469841 9803 raft_consensus.cc:2932] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.470491 6428 raft_consensus.cc:2932] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.471390 6690 raft_consensus.cc:2932] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.471896 10079 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.473160 6155 catalog_manager.cc:5484] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.475912 9189 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.482280 6691 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:12.483633 6298 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.483654 6561 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.484010 8296 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:12.484194 6625 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9772b487ca864ad98e9d79d8435c34c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.484367 8296 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.485802 9968 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.486199 6561 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.486353 6298 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.488651 6151 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:12.492090 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:12.497046 10023 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.498992 6559 consensus_queue.cc:237] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:12.499446 10026 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.500804 6428 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.501088 6496 consensus_peers.cc:485] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b9870eac6ad47d4a9ad5694dcaa1b1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.501431 10025 tablet.cc:1722] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:12.501888 6691 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.502255 8893 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.502470 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=f169430077e4414490b89649993f03ba] alter complete (version 3)
I20240503 07:59:12.503305 9963 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.505256 9277 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.505676 6691 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.505772 6428 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.507719 6154 catalog_manager.cc:5484] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.514799 6691 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:12.515911 6428 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.515946 6559 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.516211 6627 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 435f73fb9f384b10b353df19037bcac3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.516258 9347 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.516422 8077 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.517958 9968 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.518311 6428 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.520238 6559 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.520504 6156 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.527422 6428 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
W20240503 07:59:12.528745 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edf2c780d0d64db79dca88eac77d4478. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.528749 6691 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.528851 6559 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.529168 9288 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.529318 9952 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.530897 9288 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.531347 6691 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.531598 6559 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:12.533440 6156 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.542078 9958 raft_consensus.cc:1050] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:12.542387 9958 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.543485 6559 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.543951 8976 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.544000 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.544382 8895 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.545764 8895 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.545904 9958 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.547076 9963 raft_consensus.cc:1050] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:12.547421 9963 consensus_queue.cc:237] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.548645 6298 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.548739 6428 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.549008 9278 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.549090 8844 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.549335 8895 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.549717 9278 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.550971 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:12.551587 9278 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.551585 6154 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.552009 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.553735 8976 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.555562 6561 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.555634 9278 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.557699 8844 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.558912 6428 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.559986 8976 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.560021 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.560530 9952 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.561153 9952 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.561606 6428 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.561755 6151 catalog_manager.cc:5484] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.561901 6298 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.564337 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 63a31dc7bed443c196707da2988ce1a4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.564595 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a31dc7bed443c196707da2988ce1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.565196 9309 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.565323 8822 tablet_service.cc:1467] Processing CreateTablet for tablet afeff5ce33bc4205a8599bbb66b24672 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.565541 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeff5ce33bc4205a8599bbb66b24672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.566607 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 94d5216ff8654561b44a0cfd0f9b1625 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.566851 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d5216ff8654561b44a0cfd0f9b1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.566869 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 63a31dc7bed443c196707da2988ce1a4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.567111 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a31dc7bed443c196707da2988ce1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.567610 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 94d5216ff8654561b44a0cfd0f9b1625 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.567853 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d5216ff8654561b44a0cfd0f9b1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.568241 6561 consensus_queue.cc:237] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.569015 6278 tablet_service.cc:1467] Processing CreateTablet for tablet afeff5ce33bc4205a8599bbb66b24672 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.569213 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeff5ce33bc4205a8599bbb66b24672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.569233 6541 tablet_service.cc:1467] Processing CreateTablet for tablet afeff5ce33bc4205a8599bbb66b24672 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.569370 8844 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.569466 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeff5ce33bc4205a8599bbb66b24672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.569702 9963 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.570397 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5631f3e97fe04013a2a7140f44260003 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.571831 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 63a31dc7bed443c196707da2988ce1a4 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.572070 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a31dc7bed443c196707da2988ce1a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.572034 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.572156 8976 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.572544 10226 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:12.573405 10226 tablet_bootstrap.cc:654] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.574177 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 94d5216ff8654561b44a0cfd0f9b1625 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:12.574375 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d5216ff8654561b44a0cfd0f9b1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.575287 9189 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.575371 6154 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.578634 6298 raft_consensus.cc:1261] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.578792 9400 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.578904 8077 raft_consensus.cc:1050] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:12.579062 8893 consensus_queue.cc:1035] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.579245 8077 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.580163 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 5631f3e97fe04013a2a7140f44260003 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.580346 9176 tablet_replica.cc:330] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.580476 9176 raft_consensus.cc:2229] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.580612 9176 raft_consensus.cc:2258] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.580869 6561 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.581461 10227 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:12.581614 8575 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.581622 9277 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.582108 10228 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:12.582158 8296 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.582306 9968 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.583405 10230 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:12.583695 9469 raft_consensus.cc:1050] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:12.584030 10229 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:12.584053 6298 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.584164 9469 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.584420 9968 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.585505 10227 tablet_bootstrap.cc:654] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.585794 8976 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.585836 8844 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.586175 9803 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.586241 10230 tablet_bootstrap.cc:654] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.586670 10228 tablet_bootstrap.cc:654] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.587800 10229 tablet_bootstrap.cc:654] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.587824 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d95cc5335534f02aa62b00fb8b9c33c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.588133 6151 catalog_manager.cc:5484] T 5d95cc5335534f02aa62b00fb8b9c33c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.588970 6298 raft_consensus.cc:2932] T 5d95cc5335534f02aa62b00fb8b9c33c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.589143 9176 ts_tablet_manager.cc:1875] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.589780 10235 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:12.590519 6428 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.590723 10235 tablet_bootstrap.cc:654] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.591343 10229 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:12.591440 10229 ts_tablet_manager.cc:1367] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:12.591816 10229 raft_consensus.cc:348] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.591921 10229 raft_consensus.cc:374] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.588691 10230 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:12.592070 10230 ts_tablet_manager.cc:1367] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:12.592483 10230 raft_consensus.cc:348] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.592598 10230 raft_consensus.cc:374] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.592648 10230 raft_consensus.cc:729] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.592856 10230 consensus_queue.cc:260] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.593303 9803 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.593559 10235 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:12.593681 10235 ts_tablet_manager.cc:1367] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:12.594130 10235 raft_consensus.cc:348] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.594277 10235 raft_consensus.cc:374] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.594365 10235 raft_consensus.cc:729] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.594544 10235 consensus_queue.cc:260] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.594571 9968 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.596575 10235 ts_tablet_manager.cc:1398] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:12.597227 6154 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.597522 8575 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.599737 9347 raft_consensus.cc:1050] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:12.600082 9347 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.601500 6430 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.601516 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d95cc5335534f02aa62b00fb8b9c33c with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.601776 9178 tablet_replica.cc:330] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.601967 9178 raft_consensus.cc:2229] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.601979 8844 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.602169 8077 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.602327 8296 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.602708 9178 raft_consensus.cc:2258] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.604136 9178 ts_tablet_manager.cc:1875] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.605805 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.605922 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:12.605953 6691 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.606266 9178 ts_tablet_manager.cc:1888] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.606312 9178 log.cc:1196] T 5d95cc5335534f02aa62b00fb8b9c33c P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d95cc5335534f02aa62b00fb8b9c33c
I20240503 07:59:12.606735 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 5d95cc5335534f02aa62b00fb8b9c33c (table rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]) successfully deleted
I20240503 07:59:12.608950 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet aa4c954d610f423594cf9d4575f38573
I20240503 07:59:12.609477 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c954d610f423594cf9d4575f38573 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:12.609566 8575 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.609712 9796 tablet_replica.cc:330] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:12.609824 9796 raft_consensus.cc:2229] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.609923 9796 raft_consensus.cc:2258] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.609932 9968 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.610065 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c954d610f423594cf9d4575f38573 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:12.610898 9796 ts_tablet_manager.cc:1875] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.612830 9796 ts_tablet_manager.cc:1888] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.612891 9796 log.cc:1196] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/aa4c954d610f423594cf9d4575f38573
I20240503 07:59:12.613116 9796 ts_tablet_manager.cc:1909] T aa4c954d610f423594cf9d4575f38573 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:12.613091 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet aa4c954d610f423594cf9d4575f38573 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:12.613171 9968 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.614807 10259 tablet_replica.cc:330] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:12.614920 10259 raft_consensus.cc:2229] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.615052 10259 raft_consensus.cc:2258] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.615729 10259 ts_tablet_manager.cc:1875] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.615922 10258 tablet_replica.cc:330] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:12.615962 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39b01318dc1543cba2cc991e9db91c28 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.616290 10258 raft_consensus.cc:2229] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.616322 9464 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.617175 6428 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.618113 8712 raft_consensus.cc:1203] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:12.618472 10258 raft_consensus.cc:2258] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.619275 10258 ts_tablet_manager.cc:1875] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.619290 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table_new_name [id=f169430077e4414490b89649993f03ba]) successfully deleted
I20240503 07:59:12.619840 6154 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.622839 10259 ts_tablet_manager.cc:1888] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.623637 10259 log.cc:1196] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa4c954d610f423594cf9d4575f38573
I20240503 07:59:12.623601 6296 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:12.623970 10259 ts_tablet_manager.cc:1909] T aa4c954d610f423594cf9d4575f38573 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:12.624424 6155 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.624632 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table_new_name [id=f169430077e4414490b89649993f03ba]) successfully deleted
I20240503 07:59:12.624873 10258 ts_tablet_manager.cc:1888] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.625063 8575 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.624964 10258 log.cc:1196] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/aa4c954d610f423594cf9d4575f38573
I20240503 07:59:12.625797 10258 ts_tablet_manager.cc:1909] T aa4c954d610f423594cf9d4575f38573 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:12.626528 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b01318dc1543cba2cc991e9db91c28 with delete_type TABLET_DATA_TOMBSTONED (TS c167a0a433d34b74affed13cbf96e302 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.626870 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet aa4c954d610f423594cf9d4575f38573 (table rebalancer_dynamic_table_new_name [id=f169430077e4414490b89649993f03ba]) successfully deleted
I20240503 07:59:12.627754 9177 tablet_replica.cc:330] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.627851 9177 raft_consensus.cc:2229] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.628582 9177 pending_rounds.cc:62] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Trying to abort 1 pending ops.
I20240503 07:59:12.628657 9177 pending_rounds.cc:68] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023506032963567616 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "39b01318dc1543cba2cc991e9db91c28" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } } }
I20240503 07:59:12.628793 9177 raft_consensus.cc:2894] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 07:59:12.620978 8976 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.628859 9177 raft_consensus.cc:2866] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 07:59:12.628906 9177 raft_consensus.cc:2258] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.629508 10230 ts_tablet_manager.cc:1398] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240503 07:59:12.629667 10230 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:12.630026 8977 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.630828 9464 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.630913 10230 tablet_bootstrap.cc:654] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.630964 6428 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.595707 6561 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.631418 9464 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.596401 6559 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.599126 9176 ts_tablet_manager.cc:1888] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.632597 9176 log.cc:1196] T 5631f3e97fe04013a2a7140f44260003 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5631f3e97fe04013a2a7140f44260003
I20240503 07:59:12.602000 6560 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.631297 10229 raft_consensus.cc:729] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.633013 9803 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.633249 6558 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.633457 9968 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.633821 10229 consensus_queue.cc:260] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.634011 9968 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.634449 10229 ts_tablet_manager.cc:1398] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20240503 07:59:12.634883 6428 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.635344 6560 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:12.635788 6151 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.636273 10227 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:12.636292 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f943ecfe59f432c861a54b16c62572f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.636432 10227 ts_tablet_manager.cc:1367] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.000s
I20240503 07:59:12.636911 10227 raft_consensus.cc:348] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.637006 10227 raft_consensus.cc:374] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.637037 10227 raft_consensus.cc:729] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.637142 10227 consensus_queue.cc:260] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.638271 10227 ts_tablet_manager.cc:1398] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.638453 10227 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:12.639490 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 5631f3e97fe04013a2a7140f44260003 (table rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]) successfully deleted
I20240503 07:59:12.639900 10227 tablet_bootstrap.cc:654] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.640048 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f943ecfe59f432c861a54b16c62572f with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.640206 9176 tablet_replica.cc:330] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.640331 9176 raft_consensus.cc:2229] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.640455 9176 raft_consensus.cc:2258] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.640931 9176 ts_tablet_manager.cc:1875] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.641139 10228 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:12.641211 10228 ts_tablet_manager.cc:1367] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.000s
I20240503 07:59:12.641224 10230 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:12.641284 10230 ts_tablet_manager.cc:1367] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:12.641575 10230 raft_consensus.cc:348] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.641571 10228 raft_consensus.cc:348] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.641640 10230 raft_consensus.cc:374] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.641659 10230 raft_consensus.cc:729] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.641664 10228 raft_consensus.cc:374] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.641692 10228 raft_consensus.cc:729] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.641752 10230 consensus_queue.cc:260] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.642024 10230 ts_tablet_manager.cc:1398] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.641824 10228 consensus_queue.cc:260] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.642467 10228 ts_tablet_manager.cc:1398] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.642827 10227 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:12.642823 9176 ts_tablet_manager.cc:1888] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 07:59:12.642879 9176 log.cc:1196] T 2f943ecfe59f432c861a54b16c62572f P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2f943ecfe59f432c861a54b16c62572f
I20240503 07:59:12.642890 10227 ts_tablet_manager.cc:1367] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:12.643273 10227 raft_consensus.cc:348] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.643357 10227 raft_consensus.cc:374] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.643385 10227 raft_consensus.cc:729] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.643508 10227 consensus_queue.cc:260] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.643666 6151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 07:59:12.643743 10227 ts_tablet_manager.cc:1398] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.643756 6151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
W20240503 07:59:12.644753 10252 consensus_queue.cc:1159] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c167a0a433d34b74affed13cbf96e302" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240503 07:59:12.645256 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 2f943ecfe59f432c861a54b16c62572f (table rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]) successfully deleted
I20240503 07:59:12.645583 8077 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.646461 8844 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.647368 6428 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.647748 6558 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.649313 6156 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.650223 9177 ts_tablet_manager.cc:1875] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.651103 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet dc995af1acb9459f9096867bc3b075f4
I20240503 07:59:12.651326 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet 5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.651506 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet dc995af1acb9459f9096867bc3b075f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.651549 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:12.651648 9178 tablet_replica.cc:330] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.651758 9178 raft_consensus.cc:2229] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.651883 9178 raft_consensus.cc:2258] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.651919 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.652192 9177 ts_tablet_manager.cc:1888] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.652240 9177 log.cc:1196] T 39b01318dc1543cba2cc991e9db91c28 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/39b01318dc1543cba2cc991e9db91c28
I20240503 07:59:12.652302 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet dc995af1acb9459f9096867bc3b075f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.652323 9178 ts_tablet_manager.cc:1875] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.652410 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05300c9002408084f0b47d5e47591d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.652560 9177 tablet_replica.cc:330] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.652644 9177 raft_consensus.cc:2229] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.652762 9177 raft_consensus.cc:2258] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.653213 9177 ts_tablet_manager.cc:1875] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.653297 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 39b01318dc1543cba2cc991e9db91c28 (table rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]) successfully deleted
I20240503 07:59:12.654033 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05300c9002408084f0b47d5e47591d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.654249 9178 ts_tablet_manager.cc:1888] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.654307 9178 log.cc:1196] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/dc995af1acb9459f9096867bc3b075f4
I20240503 07:59:12.654256 8977 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:12.654552 9178 ts_tablet_manager.cc:1909] T dc995af1acb9459f9096867bc3b075f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:12.654711 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05300c9002408084f0b47d5e47591d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:12.654852 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:12.654914 9177 ts_tablet_manager.cc:1888] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.654971 9177 log.cc:1196] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc995af1acb9459f9096867bc3b075f4
I20240503 07:59:12.654980 9175 tablet_replica.cc:330] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.655088 9175 raft_consensus.cc:2229] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.655198 9177 ts_tablet_manager.cc:1909] T dc995af1acb9459f9096867bc3b075f4 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.655210 9175 raft_consensus.cc:2258] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.655661 9177 tablet_replica.cc:330] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.655661 9175 ts_tablet_manager.cc:1875] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.655738 9177 raft_consensus.cc:2229] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.655819 9177 raft_consensus.cc:2258] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.656111 9177 ts_tablet_manager.cc:1875] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.656997 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet dc995af1acb9459f9096867bc3b075f4 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.657610 9175 ts_tablet_manager.cc:1888] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.657665 9175 log.cc:1196] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.657888 9175 ts_tablet_manager.cc:1909] T 5f05300c9002408084f0b47d5e47591d P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:12.658427 9177 ts_tablet_manager.cc:1888] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.658480 9177 log.cc:1196] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.658694 9177 ts_tablet_manager.cc:1909] T 5f05300c9002408084f0b47d5e47591d P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.658754 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet dc995af1acb9459f9096867bc3b075f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.658878 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f05300c9002408084f0b47d5e47591d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:12.659075 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 5f05300c9002408084f0b47d5e47591d (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.659829 10259 tablet_replica.cc:330] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:12.659956 10259 raft_consensus.cc:2229] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:12.660059 10259 raft_consensus.cc:2258] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:12.660382 9177 tablet_replica.cc:330] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:12.660521 9177 raft_consensus.cc:2229] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.660997 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 610160f6ba94483a9bdcf966c413e5d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:12 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:12.661329 10258 tablet_replica.cc:330] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:12.661388 10259 ts_tablet_manager.cc:1875] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.661461 10258 raft_consensus.cc:2229] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.661569 9178 tablet_replica.cc:330] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.661571 10258 raft_consensus.cc:2258] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.661764 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet dc995af1acb9459f9096867bc3b075f4 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.661818 9178 raft_consensus.cc:2229] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.661980 9178 raft_consensus.cc:2258] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.662243 9177 raft_consensus.cc:2258] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.662761 9177 ts_tablet_manager.cc:1875] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.663231 10259 ts_tablet_manager.cc:1888] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.663305 10259 log.cc:1196] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.663535 10259 ts_tablet_manager.cc:1909] T 5f05300c9002408084f0b47d5e47591d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:12.663997 10258 ts_tablet_manager.cc:1875] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.664150 9177 ts_tablet_manager.cc:1888] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:12.664324 9177 log.cc:1196] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:12.664341 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 5f05300c9002408084f0b47d5e47591d (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.664594 9177 ts_tablet_manager.cc:1909] T 610160f6ba94483a9bdcf966c413e5d1 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:12.664552 6691 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.665253 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 610160f6ba94483a9bdcf966c413e5d1 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.665726 6558 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.665832 10258 ts_tablet_manager.cc:1888] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:12.665891 10258 log.cc:1196] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:12.665874 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 5f05300c9002408084f0b47d5e47591d (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.665932 8844 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.666126 10258 ts_tablet_manager.cc:1909] T 610160f6ba94483a9bdcf966c413e5d1 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:12.666235 8296 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.666381 9347 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.666636 9176 tablet_replica.cc:330] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.666749 9176 raft_consensus.cc:2229] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.666857 9176 raft_consensus.cc:2258] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.667073 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 610160f6ba94483a9bdcf966c413e5d1 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.667294 9176 ts_tablet_manager.cc:1875] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.667858 9347 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.668241 6558 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.669036 9176 ts_tablet_manager.cc:1888] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:12.669093 9176 log.cc:1196] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc995af1acb9459f9096867bc3b075f4
I20240503 07:59:12.669310 9176 ts_tablet_manager.cc:1909] T dc995af1acb9459f9096867bc3b075f4 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:12.669667 9178 ts_tablet_manager.cc:1875] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.669764 9176 tablet_replica.cc:330] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:12.669920 9176 raft_consensus.cc:2229] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:12.669919 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet dc995af1acb9459f9096867bc3b075f4 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.670150 9176 raft_consensus.cc:2258] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.670426 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82de209205b342cfaac10506fdac9303 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.670881 6153 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.671392 9178 ts_tablet_manager.cc:1888] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:12.671461 9178 log.cc:1196] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/610160f6ba94483a9bdcf966c413e5d1
I20240503 07:59:12.671424 9176 ts_tablet_manager.cc:1875] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:12.671685 9178 ts_tablet_manager.cc:1909] T 610160f6ba94483a9bdcf966c413e5d1 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:12.672221 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 610160f6ba94483a9bdcf966c413e5d1 (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.673192 8844 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.673434 10226 tablet_bootstrap.cc:492] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:12.673508 10226 ts_tablet_manager.cc:1367] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.000s
I20240503 07:59:12.673763 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 82de209205b342cfaac10506fdac9303 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.673911 9178 tablet_replica.cc:330] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.673910 10226 raft_consensus.cc:348] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.673995 10226 raft_consensus.cc:374] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.674015 9178 raft_consensus.cc:2229] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.674022 10226 raft_consensus.cc:729] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.674130 10226 consensus_queue.cc:260] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.674176 9178 raft_consensus.cc:2258] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.674371 10226 ts_tablet_manager.cc:1398] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.674512 10226 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:12.674850 9178 ts_tablet_manager.cc:1875] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.675247 10226 tablet_bootstrap.cc:654] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.679173 9178 ts_tablet_manager.cc:1888] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.679740 9176 ts_tablet_manager.cc:1888] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:12.679867 9178 log.cc:1196] T 82de209205b342cfaac10506fdac9303 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/82de209205b342cfaac10506fdac9303
I20240503 07:59:12.680217 9176 log.cc:1196] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f05300c9002408084f0b47d5e47591d
I20240503 07:59:12.680295 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 82de209205b342cfaac10506fdac9303 (table rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]) successfully deleted
I20240503 07:59:12.680487 9176 ts_tablet_manager.cc:1909] T 5f05300c9002408084f0b47d5e47591d P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:12.681025 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 5f05300c9002408084f0b47d5e47591d (table rebalance_test_table_10 [id=28fc17ad519645069eb8dffa88723735]) successfully deleted
I20240503 07:59:12.681478 10226 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:12.681556 10226 ts_tablet_manager.cc:1367] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:12.681916 10226 raft_consensus.cc:348] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.682003 10226 raft_consensus.cc:374] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.682034 10226 raft_consensus.cc:729] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.682132 10226 consensus_queue.cc:260] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.682368 10226 ts_tablet_manager.cc:1398] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.691969 10239 raft_consensus.cc:1050] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:12.692356 10239 consensus_queue.cc:237] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.693832 6691 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.694233 8712 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.694334 6428 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.694681 9469 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.694849 9803 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.695020 9464 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.697110 9464 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.697504 6428 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.697793 8712 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.698786 6691 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.700326 6152 catalog_manager.cc:5484] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.703758 6298 consensus_queue.cc:237] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:12.704964 6428 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.705123 8712 raft_consensus.cc:1261] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.705302 9803 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.705734 9803 consensus_queue.cc:1035] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.707080 9469 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.707597 6428 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.708282 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f417c61594348428ee9514ded49ee88 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.708484 8712 raft_consensus.cc:2932] T 7f417c61594348428ee9514ded49ee88 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:12.709923 6155 catalog_manager.cc:5484] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.711701 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:12.713537 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f417c61594348428ee9514ded49ee88 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:12.713717 9175 tablet_replica.cc:330] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:12.713840 9175 raft_consensus.cc:2229] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.713970 9175 raft_consensus.cc:2258] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.715581 9175 ts_tablet_manager.cc:1875] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.717654 9175 ts_tablet_manager.cc:1888] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.717732 9175 log.cc:1196] T 7f417c61594348428ee9514ded49ee88 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7f417c61594348428ee9514ded49ee88
I20240503 07:59:12.718209 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 7f417c61594348428ee9514ded49ee88 (table rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]) successfully deleted
I20240503 07:59:12.718815 6410 tablet_service.cc:1467] Processing CreateTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]), partition=
I20240503 07:59:12.719049 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20213b5ea4f44628d1a5f22fabd86f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.720643 10303 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:12.720832 8560 tablet_service.cc:1467] Processing CreateTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]), partition=
I20240503 07:59:12.721061 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20213b5ea4f44628d1a5f22fabd86f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.721586 10303 tablet_bootstrap.cc:654] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.722537 10303 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:12.722617 10303 ts_tablet_manager.cc:1367] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.723035 10228 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:12.723011 10303 raft_consensus.cc:348] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.723189 10303 raft_consensus.cc:374] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.723232 10303 raft_consensus.cc:729] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.723381 10303 consensus_queue.cc:260] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.723770 10228 tablet_bootstrap.cc:654] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.723798 10303 ts_tablet_manager.cc:1398] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.723980 8822 tablet_service.cc:1467] Processing CreateTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]), partition=
I20240503 07:59:12.724162 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20213b5ea4f44628d1a5f22fabd86f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.724763 10228 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:12.724862 10228 ts_tablet_manager.cc:1367] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:12.725271 10228 raft_consensus.cc:348] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.725358 10228 raft_consensus.cc:374] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.725384 10228 raft_consensus.cc:729] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.725492 10228 consensus_queue.cc:260] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.725744 10228 ts_tablet_manager.cc:1398] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:12.726039 10226 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:12.726850 10226 tablet_bootstrap.cc:654] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:12.727811 10226 tablet_bootstrap.cc:492] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:12.727890 10226 ts_tablet_manager.cc:1367] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:12.728302 10226 raft_consensus.cc:348] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.728401 10226 raft_consensus.cc:374] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:12.728432 10226 raft_consensus.cc:729] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:12.728535 10226 consensus_queue.cc:260] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.728806 10226 ts_tablet_manager.cc:1398] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.730412 6428 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47280
I20240503 07:59:12.730538 6428 raft_consensus.cc:595] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:12.741484 10000 ts_tablet_manager.cc:902] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:12.741904 10000 tablet_copy_client.cc:308] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:12.742362 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:43151)
I20240503 07:59:12.742480 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 0e3f58a27f2a4a2593e8219b1d7bd894 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:43151: session id = 9553fe3c4ccd4e91b7c176aad82fec09-0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:12.743513 6581 tablet_copy_source_session.cc:205] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.744379 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f58a27f2a4a2593e8219b1d7bd894. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.746424 10000 tablet_copy_client.cc:783] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.746564 10000 tablet_copy_client.cc:647] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.747552 10000 tablet_copy_client.cc:523] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.748641 10000 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:12.754341 9952 raft_consensus.cc:482] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.754443 9952 raft_consensus.cc:504] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.754750 9952 leader_election.cc:288] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:12.755040 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20213b5ea4f44628d1a5f22fabd86f4" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:12.755143 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20213b5ea4f44628d1a5f22fabd86f4" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:12.755174 8575 raft_consensus.cc:2454] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:12.755254 8844 raft_consensus.cc:2454] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:12.755524 6365 leader_election.cc:302] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:12.755875 9952 raft_consensus.cc:2781] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.755939 9952 raft_consensus.cc:482] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.755976 9952 raft_consensus.cc:3037] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.757036 9952 raft_consensus.cc:504] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.757350 9952 leader_election.cc:288] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:12.757606 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20213b5ea4f44628d1a5f22fabd86f4" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:12.757706 8575 raft_consensus.cc:3037] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.757700 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20213b5ea4f44628d1a5f22fabd86f4" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:12.757779 8844 raft_consensus.cc:3037] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.758507 8844 raft_consensus.cc:2454] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:12.758507 8575 raft_consensus.cc:2454] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:12.758788 6365 leader_election.cc:302] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:12.758992 9952 raft_consensus.cc:2781] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.759063 9952 raft_consensus.cc:686] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:12.759178 9952 consensus_queue.cc:237] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:12.760764 9464 raft_consensus.cc:1050] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:12.761189 9464 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.761139 6152 catalog_manager.cc:5484] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.762548 6428 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.762560 6691 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.762923 10239 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.763103 9469 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.763500 8844 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:12.765363 10239 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:12.766322 10239 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.768743 6428 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.769127 6152 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.770462 6691 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.771462 8844 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.775162 6298 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:12.776299 8844 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.776619 6691 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:12.776654 10000 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.776939 10000 tablet_bootstrap.cc:492] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:12.776970 9464 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.777132 10000 ts_tablet_manager.cc:1367] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.004s
I20240503 07:59:12.777158 9803 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:12.777519 10000 raft_consensus.cc:348] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.777648 10000 raft_consensus.cc:729] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:12.777746 10000 consensus_queue.cc:260] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.778007 10000 ts_tablet_manager.cc:1398] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.778573 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-0e3f58a27f2a4a2593e8219b1d7bd894 received from {username='slave'} at 127.0.99.134:43151
I20240503 07:59:12.778640 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-0e3f58a27f2a4a2593e8219b1d7bd894 on tablet 0e3f58a27f2a4a2593e8219b1d7bd894 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:12.778921 9464 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.779300 6691 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.779300 8844 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.781225 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4308bc87be24f28b6566ce76cae6e50 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:12.781893 6152 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:12.784610 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet d4308bc87be24f28b6566ce76cae6e50 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:12.784941 9178 tablet_replica.cc:330] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:12.785089 9178 raft_consensus.cc:2229] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:12.785255 9178 raft_consensus.cc:2258] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:12.785876 9178 ts_tablet_manager.cc:1875] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:12.788038 9178 ts_tablet_manager.cc:1888] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:12.788103 9178 log.cc:1196] T d4308bc87be24f28b6566ce76cae6e50 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4308bc87be24f28b6566ce76cae6e50
I20240503 07:59:12.788515 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet d4308bc87be24f28b6566ce76cae6e50 (table rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]) successfully deleted
I20240503 07:59:12.792356 9952 raft_consensus.cc:979] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: : Instructing follower ad910f92ef9d4db8a8962cecba289f8a to start an election
I20240503 07:59:12.792451 9952 raft_consensus.cc:1067] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Signalling peer ad910f92ef9d4db8a8962cecba289f8a to start an election
I20240503 07:59:12.793133 6558 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa"
dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
from {username='slave'} at 127.0.99.130:51161
I20240503 07:59:12.793283 6558 raft_consensus.cc:482] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:12.793354 6558 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:12.794157 6558 raft_consensus.cc:504] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.794546 6558 leader_election.cc:288] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:12.794754 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:12.794857 6298 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:12.794826 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:12.794931 6428 raft_consensus.cc:3032] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:12.794968 6428 raft_consensus.cc:729] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:12.794935 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e3555fb9214a40bd8dbf937b87cbaa" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:12.795029 8575 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:12.795110 6428 consensus_queue.cc:260] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.795277 6428 raft_consensus.cc:3037] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:12.795737 6298 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:12.796000 8575 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:12.796034 6428 raft_consensus.cc:2454] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:12.796317 6494 leader_election.cc:302] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:12.796501 9278 raft_consensus.cc:2781] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:12.796577 9278 raft_consensus.cc:686] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 2 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:12.796702 9278 consensus_queue.cc:237] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:12.798601 6155 catalog_manager.cc:5484] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 1 to 2, leader changed from d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.804701 6691 consensus_queue.cc:237] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:12.805917 6298 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.805975 6558 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.806298 6627 consensus_peers.cc:485] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2762b2e6de842bb8c898aea5b3910ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.806380 8296 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.806674 8296 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.807909 8296 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.808480 6298 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.808547 6558 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:12.810456 6155 catalog_manager.cc:5484] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.811533 10370 ts_tablet_manager.cc:902] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:12.812284 10370 tablet_copy_client.cc:308] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:12.812860 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 7d855e5edf05471983dfdbd40f37560f from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:60775)
I20240503 07:59:12.812943 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 7d855e5edf05471983dfdbd40f37560f from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:60775: session id = c8d71787690245e681c85a087086e7ed-7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:12.813756 6581 tablet_copy_source_session.cc:205] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.814569 10370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d855e5edf05471983dfdbd40f37560f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.816099 10370 tablet_copy_client.cc:783] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.816222 10370 tablet_copy_client.cc:647] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.817133 10370 tablet_copy_client.cc:523] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.817986 10370 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:12.822564 6558 consensus_queue.cc:237] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:12.823660 6428 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:12.823714 6691 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:12.823928 6496 consensus_peers.cc:485] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 824c42be8152494fad2cbc76ece56981. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:12.824012 9963 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.824131 9278 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:12.825706 9278 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.826087 6691 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.828223 6152 catalog_manager.cc:5484] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.833947 6428 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:12.839321 10095 raft_consensus.cc:482] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.839432 10095 raft_consensus.cc:504] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.839783 10095 leader_election.cc:288] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:12.840103 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a31dc7bed443c196707da2988ce1a4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:12.840222 8977 raft_consensus.cc:2454] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:12.840492 8778 leader_election.cc:302] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:12.840484 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a31dc7bed443c196707da2988ce1a4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:12.840607 8712 raft_consensus.cc:2454] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:12.840651 10095 raft_consensus.cc:2781] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.840695 10095 raft_consensus.cc:482] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.840734 10095 raft_consensus.cc:3037] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.841946 10095 raft_consensus.cc:504] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.842267 10095 leader_election.cc:288] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:12.842535 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a31dc7bed443c196707da2988ce1a4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:12.842633 8712 raft_consensus.cc:3037] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.843487 8712 raft_consensus.cc:2454] T 63a31dc7bed443c196707da2988ce1a4 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:12.843690 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a31dc7bed443c196707da2988ce1a4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:12.843789 8977 raft_consensus.cc:3037] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.844544 8977 raft_consensus.cc:2454] T 63a31dc7bed443c196707da2988ce1a4 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:12.845089 8778 leader_election.cc:302] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:12.845213 10000 ts_tablet_manager.cc:902] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:12.845609 10000 tablet_copy_client.cc:308] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:12.845777 10095 raft_consensus.cc:2781] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.845860 10095 raft_consensus.cc:686] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:12.845970 10095 consensus_queue.cc:237] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.846012 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 11fbfb84cef74316a77b3ccfab49a19b from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:32897)
I20240503 07:59:12.846093 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 11fbfb84cef74316a77b3ccfab49a19b from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:32897: session id = 9553fe3c4ccd4e91b7c176aad82fec09-11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:12.847046 6318 tablet_copy_source_session.cc:205] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.847637 6153 catalog_manager.cc:5484] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:12.848349 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fbfb84cef74316a77b3ccfab49a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.850227 10000 tablet_copy_client.cc:783] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.850371 10000 tablet_copy_client.cc:647] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.851282 10000 tablet_copy_client.cc:523] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.852169 10000 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:12.861552 8893 raft_consensus.cc:482] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.861640 8893 raft_consensus.cc:504] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.861966 8893 leader_election.cc:288] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:12.862712 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff5ce33bc4205a8599bbb66b24672" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:12.862826 6298 raft_consensus.cc:2454] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:12.863013 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff5ce33bc4205a8599bbb66b24672" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:12.863109 8843 raft_consensus.cc:2454] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:12.863456 6496 leader_election.cc:302] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:12.864055 8893 raft_consensus.cc:2781] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.864106 8893 raft_consensus.cc:482] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.864137 8893 raft_consensus.cc:3037] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.864224 10401 ts_tablet_manager.cc:902] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:12.864944 10401 tablet_copy_client.cc:308] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:12.865495 8893 raft_consensus.cc:504] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.865805 8893 leader_election.cc:288] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:12.866101 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet f5604af852814da58385f072eb9600aa from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:58189)
I20240503 07:59:12.866221 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet f5604af852814da58385f072eb9600aa from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:58189: session id = 616fdda6ba714cb7be01a564de2b9db2-f5604af852814da58385f072eb9600aa
I20240503 07:59:12.866520 8844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff5ce33bc4205a8599bbb66b24672" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:12.866611 8844 raft_consensus.cc:3037] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.867143 6450 tablet_copy_source_session.cc:205] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.867913 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5604af852814da58385f072eb9600aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.869438 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff5ce33bc4205a8599bbb66b24672" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:12.869536 6298 raft_consensus.cc:3037] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.869788 10401 tablet_copy_client.cc:783] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.869932 10401 tablet_copy_client.cc:647] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.870366 6298 raft_consensus.cc:2454] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:12.870695 6494 leader_election.cc:302] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:12.870945 8893 raft_consensus.cc:2781] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.871022 8893 raft_consensus.cc:686] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:12.871137 8893 consensus_queue.cc:237] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:12.871204 8844 raft_consensus.cc:2454] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:12.873075 6152 catalog_manager.cc:5484] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.874245 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:12.876922 10401 tablet_copy_client.cc:523] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.878093 10401 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:12.878228 8844 raft_consensus.cc:1261] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.878579 9958 consensus_queue.cc:1035] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.879210 8575 raft_consensus.cc:1261] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:12.880338 9958 consensus_queue.cc:1035] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:12.880708 6152 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]). Expected version 1 got 0
I20240503 07:59:12.882566 6153 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]). Expected version 1 got 0
I20240503 07:59:12.882910 10370 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.883164 10370 tablet_bootstrap.cc:492] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:12.883356 10370 ts_tablet_manager.cc:1367] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.000s
I20240503 07:59:12.883800 10370 raft_consensus.cc:348] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:12.884212 10370 raft_consensus.cc:729] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:12.884311 10370 consensus_queue.cc:260] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:12.884392 10026 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.884624 10026 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:12.884649 10370 ts_tablet_manager.cc:1398] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.885536 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-7d855e5edf05471983dfdbd40f37560f received from {username='slave'} at 127.0.99.133:60775
I20240503 07:59:12.885597 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session c8d71787690245e681c85a087086e7ed-7d855e5edf05471983dfdbd40f37560f on tablet 7d855e5edf05471983dfdbd40f37560f with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:12.891809 6152 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]). Expected version 1 got 0
I20240503 07:59:12.893316 10023 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.893577 10023 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:12.894940 9989 raft_consensus.cc:482] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:12.895040 9989 raft_consensus.cc:504] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.895489 9989 leader_election.cc:288] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:12.895922 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d5216ff8654561b44a0cfd0f9b1625" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:12.896039 8575 raft_consensus.cc:2454] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:12.896253 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d5216ff8654561b44a0cfd0f9b1625" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:12.896371 6298 raft_consensus.cc:2454] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:12.897089 8910 leader_election.cc:302] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:12.897991 9989 raft_consensus.cc:2781] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:12.898061 9989 raft_consensus.cc:482] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:12.898106 9989 raft_consensus.cc:3037] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.899068 9989 raft_consensus.cc:504] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.899266 6155 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1]). Expected version 1 got 0
I20240503 07:59:12.899403 9989 leader_election.cc:288] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:12.899710 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d5216ff8654561b44a0cfd0f9b1625" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:12.899720 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d5216ff8654561b44a0cfd0f9b1625" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:12.899809 6298 raft_consensus.cc:3037] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.899813 8575 raft_consensus.cc:3037] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:12.900182 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1] alter complete (version 1)
I20240503 07:59:12.900653 8575 raft_consensus.cc:2454] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:12.900750 6298 raft_consensus.cc:2454] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:12.901239 8910 leader_election.cc:302] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:12.901620 9989 raft_consensus.cc:2781] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:12.901698 9989 raft_consensus.cc:686] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:12.901811 9989 consensus_queue.cc:237] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:12.903523 6155 catalog_manager.cc:5484] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:12.904613 10000 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.904933 10000 tablet_bootstrap.cc:492] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:12.905150 10000 ts_tablet_manager.cc:1367] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.053s user 0.020s sys 0.004s
I20240503 07:59:12.905562 10000 raft_consensus.cc:348] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.905660 10000 raft_consensus.cc:729] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:12.905748 10000 consensus_queue.cc:260] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:12.905797 10411 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:12.906117 10000 ts_tablet_manager.cc:1398] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.906131 10411 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:12.906637 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-11fbfb84cef74316a77b3ccfab49a19b received from {username='slave'} at 127.0.99.134:32897
I20240503 07:59:12.906705 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-11fbfb84cef74316a77b3ccfab49a19b on tablet 11fbfb84cef74316a77b3ccfab49a19b with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:12.919953 10414 ts_tablet_manager.cc:902] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:12.920531 10414 tablet_copy_client.cc:308] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:12.920964 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 8c01e3ce57874e24ae75a41243fdeaaf from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:53021)
I20240503 07:59:12.921052 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 8c01e3ce57874e24ae75a41243fdeaaf from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:53021: session id = 3d08a06d53b14381bee6223c4e1a6027-8c01e3ce57874e24ae75a41243fdeaaf
I20240503 07:59:12.921720 6450 tablet_copy_source_session.cc:205] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.922549 10414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c01e3ce57874e24ae75a41243fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.924947 10414 tablet_copy_client.cc:783] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.925110 10414 tablet_copy_client.cc:647] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.926275 10414 tablet_copy_client.cc:523] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.927408 10414 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:12.928262 10401 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.928550 10401 tablet_bootstrap.cc:492] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:12.928838 10401 ts_tablet_manager.cc:1367] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.051s user 0.026s sys 0.001s
I20240503 07:59:12.929350 10401 raft_consensus.cc:348] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:12.929522 10401 raft_consensus.cc:729] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:12.929657 10401 consensus_queue.cc:260] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:12.930590 10401 ts_tablet_manager.cc:1398] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:12.931043 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-f5604af852814da58385f072eb9600aa received from {username='slave'} at 127.0.99.135:58189
I20240503 07:59:12.931109 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-f5604af852814da58385f072eb9600aa on tablet f5604af852814da58385f072eb9600aa with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:12.958092 10414 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:12.958401 10414 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:12.958616 10414 ts_tablet_manager.cc:1367] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20240503 07:59:12.959277 10414 raft_consensus.cc:348] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:12.959455 10414 raft_consensus.cc:729] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:12.959618 10414 consensus_queue.cc:260] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:12.959955 10414 ts_tablet_manager.cc:1398] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:12.960599 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-8c01e3ce57874e24ae75a41243fdeaaf received from {username='slave'} at 127.0.99.136:53021
I20240503 07:59:12.960664 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-8c01e3ce57874e24ae75a41243fdeaaf on tablet 8c01e3ce57874e24ae75a41243fdeaaf with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:12.986321 10401 ts_tablet_manager.cc:902] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:12.986747 10401 tablet_copy_client.cc:308] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:12.987174 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 435f73fb9f384b10b353df19037bcac3 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:12.987279 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 435f73fb9f384b10b353df19037bcac3 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-435f73fb9f384b10b353df19037bcac3
I20240503 07:59:12.988034 6712 tablet_copy_source_session.cc:205] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:12.989411 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f73fb9f384b10b353df19037bcac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:12.992385 10401 tablet_copy_client.cc:783] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:12.992513 10401 tablet_copy_client.cc:647] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:12.993968 8692 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:12.995914 10401 tablet_copy_client.cc:523] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:12.997020 10401 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.002766 8957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.004247 6277 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:13.004974 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:13.010488 10411 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.010766 10411 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:13.011389 10023 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.011617 10023 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:13.011579 10026 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.011837 10026 tablet.cc:1620] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:13.015961 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4fb32b35f0174857883fd44a88f4dfc1] alter complete (version 2)
I20240503 07:59:13.018882 6672 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 07:59:13.029158 8822 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:13.036298 6410 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:13.040650 6539 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:13.054644 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:13.069597 10414 ts_tablet_manager.cc:902] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:13.070035 10414 tablet_copy_client.cc:308] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:13.070472 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:36739)
I20240503 07:59:13.070571 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:36739: session id = 3d08a06d53b14381bee6223c4e1a6027-4b9870eac6ad47d4a9ad5694dcaa1b1f
I20240503 07:59:13.072306 6581 tablet_copy_source_session.cc:205] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.073150 10414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.075271 10414 tablet_copy_client.cc:783] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.075410 10414 tablet_copy_client.cc:647] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.076972 10414 tablet_copy_client.cc:523] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.078094 10414 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:13.085551 10401 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.086367 10401 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:13.086519 10401 ts_tablet_manager.cc:1367] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.090s user 0.023s sys 0.000s
I20240503 07:59:13.091142 10401 raft_consensus.cc:348] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.091276 10401 raft_consensus.cc:729] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:13.091377 10401 consensus_queue.cc:260] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.091653 10401 ts_tablet_manager.cc:1398] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:13.092396 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-435f73fb9f384b10b353df19037bcac3 received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:13.092464 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-435f73fb9f384b10b353df19037bcac3 on tablet 435f73fb9f384b10b353df19037bcac3 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:13.093123 10370 ts_tablet_manager.cc:902] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:13.093562 10370 tablet_copy_client.cc:308] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:13.094115 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 9772b487ca864ad98e9d79d8435c34c6 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:54869)
I20240503 07:59:13.094203 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 9772b487ca864ad98e9d79d8435c34c6 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:54869: session id = c8d71787690245e681c85a087086e7ed-9772b487ca864ad98e9d79d8435c34c6
I20240503 07:59:13.095011 6712 tablet_copy_source_session.cc:205] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.095952 10370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9772b487ca864ad98e9d79d8435c34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.097936 10370 tablet_copy_client.cc:783] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.098049 10370 tablet_copy_client.cc:647] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.100049 10370 tablet_copy_client.cc:523] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.101051 10370 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:13.103582 10414 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.103796 10414 tablet_bootstrap.cc:492] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:13.103960 10414 ts_tablet_manager.cc:1367] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.000s
I20240503 07:59:13.104360 10414 raft_consensus.cc:348] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.104486 10414 raft_consensus.cc:729] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:13.104586 10414 consensus_queue.cc:260] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.104890 10414 ts_tablet_manager.cc:1398] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:13.105544 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-4b9870eac6ad47d4a9ad5694dcaa1b1f received from {username='slave'} at 127.0.99.136:36739
I20240503 07:59:13.105620 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-4b9870eac6ad47d4a9ad5694dcaa1b1f on tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:13.109292 10000 ts_tablet_manager.cc:902] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:13.109685 10000 tablet_copy_client.cc:308] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:13.110087 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet edf2c780d0d64db79dca88eac77d4478 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:40145)
I20240503 07:59:13.110178 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet edf2c780d0d64db79dca88eac77d4478 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:40145: session id = 9553fe3c4ccd4e91b7c176aad82fec09-edf2c780d0d64db79dca88eac77d4478
I20240503 07:59:13.110996 6450 tablet_copy_source_session.cc:205] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.111768 10000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf2c780d0d64db79dca88eac77d4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.113715 10000 tablet_copy_client.cc:783] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.113831 10000 tablet_copy_client.cc:647] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.114733 10000 tablet_copy_client.cc:523] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.115684 10000 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:13.118459 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:13.124756 10411 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.125180 10026 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.126377 10023 tablet.cc:1722] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.131443 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=4fb32b35f0174857883fd44a88f4dfc1] alter complete (version 3)
I20240503 07:59:13.135839 6691 consensus_queue.cc:237] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.136737 10370 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.137015 10370 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:13.137182 10370 ts_tablet_manager.cc:1367] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.003s
I20240503 07:59:13.137521 6558 raft_consensus.cc:1261] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:13.137630 6625 consensus_peers.cc:485] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b29dafc7b564493a8fdf064e7d3461ef. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.137693 6298 raft_consensus.cc:1261] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.138072 10370 raft_consensus.cc:348] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.138233 10370 raft_consensus.cc:729] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:13.138268 8296 consensus_queue.cc:1035] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.138382 10370 consensus_queue.cc:260] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.138561 8296 consensus_queue.cc:1035] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.139189 10370 ts_tablet_manager.cc:1398] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:13.139851 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-9772b487ca864ad98e9d79d8435c34c6 received from {username='slave'} at 127.0.99.133:54869
I20240503 07:59:13.139919 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session c8d71787690245e681c85a087086e7ed-9772b487ca864ad98e9d79d8435c34c6 on tablet 9772b487ca864ad98e9d79d8435c34c6 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:13.140388 10254 raft_consensus.cc:2932] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.142268 6298 raft_consensus.cc:2932] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.142532 6558 raft_consensus.cc:2932] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.143260 10000 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.143512 10000 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:13.143672 10000 ts_tablet_manager.cc:1367] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.000s
I20240503 07:59:13.144120 10000 raft_consensus.cc:348] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:13.144224 10000 raft_consensus.cc:729] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:13.144312 10000 consensus_queue.cc:260] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:13.144699 10000 ts_tablet_manager.cc:1398] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.145323 6150 catalog_manager.cc:5484] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.146965 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-edf2c780d0d64db79dca88eac77d4478 received from {username='slave'} at 127.0.99.134:40145
I20240503 07:59:13.147023 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-edf2c780d0d64db79dca88eac77d4478 on tablet edf2c780d0d64db79dca88eac77d4478 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:13.154130 6151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 07:59:13.154232 6151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 07:59:13.154951 6691 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
W20240503 07:59:13.156289 6627 consensus_peers.cc:485] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8861b52c306e43d8b0615202cf7d8e35. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.156391 6428 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.156440 6558 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.156809 9968 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.156996 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet 11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:13.157002 8296 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.157311 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 18677aa766454b8c98a1a2f4a48feee2
I20240503 07:59:13.157537 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.157557 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet b29dafc7b564493a8fdf064e7d3461ef
I20240503 07:59:13.157688 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.157778 9177 tablet_replica.cc:330] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.157822 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:13.157923 9177 raft_consensus.cc:2229] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.157963 10259 tablet_replica.cc:330] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:13.158077 10259 raft_consensus.cc:2229] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:13.158156 10259 raft_consensus.cc:2258] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:13.158149 9177 raft_consensus.cc:2258] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.158229 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbfb84cef74316a77b3ccfab49a19b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:13.158316 9178 tablet_replica.cc:330] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.158434 9178 raft_consensus.cc:2229] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.158495 9175 tablet_replica.cc:330] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:13.158577 9178 raft_consensus.cc:2258] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.158601 9175 raft_consensus.cc:2229] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.158666 10259 ts_tablet_manager.cc:1875] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.158721 9175 raft_consensus.cc:2258] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.158726 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet b29dafc7b564493a8fdf064e7d3461ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:13.159382 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.159631 9175 ts_tablet_manager.cc:1875] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.159674 9178 ts_tablet_manager.cc:1875] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.160319 10259 ts_tablet_manager.cc:1888] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.160372 10259 log.cc:1196] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:13.160531 10259 ts_tablet_manager.cc:1909] T 11fbfb84cef74316a77b3ccfab49a19b P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:13.161242 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.161316 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet b29dafc7b564493a8fdf064e7d3461ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.161571 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet b29dafc7b564493a8fdf064e7d3461ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:13.161712 9178 ts_tablet_manager.cc:1888] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.161768 9178 log.cc:1196] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:13.162007 9178 ts_tablet_manager.cc:1909] T 11fbfb84cef74316a77b3ccfab49a19b P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:13.162055 9968 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:13.162268 9177 ts_tablet_manager.cc:1875] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.162763 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet b29dafc7b564493a8fdf064e7d3461ef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:13.162928 9176 tablet_replica.cc:330] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:13.163053 9176 raft_consensus.cc:2229] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.163182 9176 raft_consensus.cc:2258] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.162940 9178 tablet_replica.cc:330] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.163385 9178 raft_consensus.cc:2229] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.163542 9178 raft_consensus.cc:2258] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.163767 9176 ts_tablet_manager.cc:1875] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.164062 9178 ts_tablet_manager.cc:1875] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.164096 6428 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:13.164315 9177 ts_tablet_manager.cc:1888] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.164398 9177 log.cc:1196] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:13.164629 9177 ts_tablet_manager.cc:1909] T 11fbfb84cef74316a77b3ccfab49a19b P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.164844 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 18677aa766454b8c98a1a2f4a48feee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:13.165097 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 11fbfb84cef74316a77b3ccfab49a19b (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.165819 9176 ts_tablet_manager.cc:1888] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.165902 9176 log.cc:1196] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b29dafc7b564493a8fdf064e7d3461ef
I20240503 07:59:13.166142 9176 ts_tablet_manager.cc:1909] T b29dafc7b564493a8fdf064e7d3461ef P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:13.166255 9175 ts_tablet_manager.cc:1888] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.166316 9175 log.cc:1196] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/11fbfb84cef74316a77b3ccfab49a19b
I20240503 07:59:13.166569 9175 ts_tablet_manager.cc:1909] T 11fbfb84cef74316a77b3ccfab49a19b P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:13.166746 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet b29dafc7b564493a8fdf064e7d3461ef (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.166854 6151 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.167135 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 11fbfb84cef74316a77b3ccfab49a19b (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.167093 6558 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:13.167713 9176 tablet_replica.cc:330] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:13.167825 9176 raft_consensus.cc:2229] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.167927 9176 raft_consensus.cc:2258] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.168349 9176 ts_tablet_manager.cc:1875] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.168568 9175 tablet_replica.cc:330] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:13.168677 9175 raft_consensus.cc:2229] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.168885 9175 raft_consensus.cc:2258] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.168870 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 11fbfb84cef74316a77b3ccfab49a19b (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.169137 9177 tablet_replica.cc:330] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.169243 9177 raft_consensus.cc:2229] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.169703 9177 raft_consensus.cc:2258] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.170120 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 11fbfb84cef74316a77b3ccfab49a19b (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.170172 9177 ts_tablet_manager.cc:1875] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.171962 9177 ts_tablet_manager.cc:1888] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.172034 9177 log.cc:1196] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/18677aa766454b8c98a1a2f4a48feee2
I20240503 07:59:13.172060 9176 ts_tablet_manager.cc:1888] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.172137 9176 log.cc:1196] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/18677aa766454b8c98a1a2f4a48feee2
I20240503 07:59:13.172247 9177 ts_tablet_manager.cc:1909] T 18677aa766454b8c98a1a2f4a48feee2 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.172382 9176 ts_tablet_manager.cc:1909] T 18677aa766454b8c98a1a2f4a48feee2 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:13.172871 9177 tablet_replica.cc:330] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.172850 9178 ts_tablet_manager.cc:1888] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.172930 9178 log.cc:1196] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/18677aa766454b8c98a1a2f4a48feee2
I20240503 07:59:13.172927 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 18677aa766454b8c98a1a2f4a48feee2 (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.173036 9177 raft_consensus.cc:2229] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.173069 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 18677aa766454b8c98a1a2f4a48feee2 (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.173194 9177 raft_consensus.cc:2258] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.173478 9178 ts_tablet_manager.cc:1909] T 18677aa766454b8c98a1a2f4a48feee2 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:13.173741 9175 ts_tablet_manager.cc:1875] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.174089 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 18677aa766454b8c98a1a2f4a48feee2 (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
W20240503 07:59:13.174844 6141 catalog_manager.cc:4797] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): delete failed for tablet b29dafc7b564493a8fdf064e7d3461ef because the tablet was not found. No further retry: Not found: Tablet not found: b29dafc7b564493a8fdf064e7d3461ef
I20240503 07:59:13.175410 9175 ts_tablet_manager.cc:1888] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.175467 9175 log.cc:1196] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b29dafc7b564493a8fdf064e7d3461ef
I20240503 07:59:13.175644 9175 ts_tablet_manager.cc:1909] T b29dafc7b564493a8fdf064e7d3461ef P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:13.176360 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet b29dafc7b564493a8fdf064e7d3461ef (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.176414 9177 ts_tablet_manager.cc:1875] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.177742 9177 ts_tablet_manager.cc:1888] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.177799 9177 log.cc:1196] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b29dafc7b564493a8fdf064e7d3461ef
I20240503 07:59:13.177973 9177 ts_tablet_manager.cc:1909] T b29dafc7b564493a8fdf064e7d3461ef P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.178565 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet b29dafc7b564493a8fdf064e7d3461ef (table rebalance_test_table_12 [id=280837395f114a969bd89f792265a5e2]) successfully deleted
I20240503 07:59:13.186941 8977 consensus_queue.cc:237] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.188352 8712 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.188539 8575 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.188810 10502 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.189478 10502 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.191200 9196 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.191553 8712 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.191553 8575 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.194196 6151 catalog_manager.cc:5484] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
W20240503 07:59:13.196127 8912 consensus_peers.cc:485] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a01fa99be75f498a8394aabbfabb3f5a. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.199911 8575 raft_consensus.cc:1203] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.204480 6298 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.205772 8977 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:13.205790 6232 consensus_peers.cc:485] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f943ecfe59f432c861a54b16c62572f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.205849 6428 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:13.206128 10239 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.206297 9803 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.207748 9464 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.208158 8977 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.208204 6428 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.210896 6157 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.216629 6428 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:13.217113 9963 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.235335 6151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.235448 6151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.237169 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet e20213b5ea4f44628d1a5f22fabd86f4
I20240503 07:59:13.237742 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.237962 9178 tablet_replica.cc:330] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.238021 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:13.238086 9178 raft_consensus.cc:2229] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.238175 10495 tablet_replica.cc:330] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:13.238185 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet e20213b5ea4f44628d1a5f22fabd86f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:13.238240 9178 raft_consensus.cc:2258] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.238291 10495 raft_consensus.cc:2229] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.238415 10495 raft_consensus.cc:2258] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.238996 10495 ts_tablet_manager.cc:1875] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.238996 9178 ts_tablet_manager.cc:1875] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.240965 9178 ts_tablet_manager.cc:1888] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.241029 9178 log.cc:1196] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e20213b5ea4f44628d1a5f22fabd86f4
I20240503 07:59:13.241248 9178 ts_tablet_manager.cc:1909] T e20213b5ea4f44628d1a5f22fabd86f4 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:13.241531 10495 ts_tablet_manager.cc:1888] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.241586 10495 log.cc:1196] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e20213b5ea4f44628d1a5f22fabd86f4
I20240503 07:59:13.241807 10495 ts_tablet_manager.cc:1909] T e20213b5ea4f44628d1a5f22fabd86f4 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:13.241935 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table_new_name [id=4fb32b35f0174857883fd44a88f4dfc1]) successfully deleted
I20240503 07:59:13.242405 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table_new_name [id=4fb32b35f0174857883fd44a88f4dfc1]) successfully deleted
I20240503 07:59:13.244908 10522 tablet_replica.cc:330] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:13.245129 10522 raft_consensus.cc:2229] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.245254 10522 raft_consensus.cc:2258] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.245862 10522 ts_tablet_manager.cc:1875] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.247864 10522 ts_tablet_manager.cc:1888] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.247936 10522 log.cc:1196] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/e20213b5ea4f44628d1a5f22fabd86f4
I20240503 07:59:13.248203 10522 ts_tablet_manager.cc:1909] T e20213b5ea4f44628d1a5f22fabd86f4 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:13.248741 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet e20213b5ea4f44628d1a5f22fabd86f4 (table rebalancer_dynamic_table_new_name [id=4fb32b35f0174857883fd44a88f4dfc1]) successfully deleted
I20240503 07:59:13.257088 8575 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:13.257465 9963 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.260072 6298 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:13.260598 9278 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.270885 8712 raft_consensus.cc:1203] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.293453 10502 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.296587 10502 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.309543 8957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:13.317232 8822 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 07:59:13.318786 9317 consensus_queue.cc:1035] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.320039 6670 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:13.321985 8692 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:13.328704 6539 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:13.332639 6279 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 07:59:13.344074 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:13.346206 8844 raft_consensus.cc:1203] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.353579 9278 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.355108 6410 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:13.357597 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]), partition=
I20240503 07:59:13.357836 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de1f1d3f05d42b39563d6cb5857f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.357991 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]), partition=
I20240503 07:59:13.358191 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de1f1d3f05d42b39563d6cb5857f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.359972 10564 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.360108 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]), partition=
I20240503 07:59:13.360340 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de1f1d3f05d42b39563d6cb5857f7c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.360750 10564 tablet_bootstrap.cc:654] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.362320 10567 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:13.363168 8560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 07:59:13.363572 10567 tablet_bootstrap.cc:654] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.364878 10567 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:13.364959 10567 ts_tablet_manager.cc:1367] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:13.365410 10567 raft_consensus.cc:348] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.365496 10567 raft_consensus.cc:374] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.365522 10567 raft_consensus.cc:729] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.365628 10567 consensus_queue.cc:260] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.365871 10567 ts_tablet_manager.cc:1398] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.369472 10565 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:13.370332 10565 tablet_bootstrap.cc:654] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.371678 10095 consensus_queue.cc:1035] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.374912 10401 ts_tablet_manager.cc:902] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:13.375389 10401 tablet_copy_client.cc:308] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:13.375720 10564 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:13.375819 10564 ts_tablet_manager.cc:1367] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:13.376144 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet 824c42be8152494fad2cbc76ece56981 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:33533)
I20240503 07:59:13.376183 10564 raft_consensus.cc:348] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.376243 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet 824c42be8152494fad2cbc76ece56981 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:33533: session id = 616fdda6ba714cb7be01a564de2b9db2-824c42be8152494fad2cbc76ece56981
I20240503 07:59:13.376263 10564 raft_consensus.cc:374] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.376291 10564 raft_consensus.cc:729] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.376420 10564 consensus_queue.cc:260] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.376667 10564 ts_tablet_manager.cc:1398] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.377161 6581 tablet_copy_source_session.cc:205] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.377625 10565 tablet_bootstrap.cc:492] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:13.377707 10565 ts_tablet_manager.cc:1367] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:13.378090 10565 raft_consensus.cc:348] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.378196 10565 raft_consensus.cc:374] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.378230 10565 raft_consensus.cc:729] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.378356 10565 consensus_queue.cc:260] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.378592 10565 ts_tablet_manager.cc:1398] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.381482 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c42be8152494fad2cbc76ece56981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.383504 10401 tablet_copy_client.cc:783] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.383621 10401 tablet_copy_client.cc:647] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.393862 10401 tablet_copy_client.cc:523] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.395005 10401 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.398293 9277 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.405382 10414 ts_tablet_manager.cc:902] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:13.405848 10414 tablet_copy_client.cc:308] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:13.410104 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet a2762b2e6de842bb8c898aea5b3910ba from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:36575)
I20240503 07:59:13.410205 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet a2762b2e6de842bb8c898aea5b3910ba from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:36575: session id = 3d08a06d53b14381bee6223c4e1a6027-a2762b2e6de842bb8c898aea5b3910ba
I20240503 07:59:13.411072 6712 tablet_copy_source_session.cc:205] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.412194 8977 raft_consensus.cc:1203] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.412324 10414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2762b2e6de842bb8c898aea5b3910ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.414371 10414 tablet_copy_client.cc:783] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.414498 10414 tablet_copy_client.cc:647] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.416757 10414 tablet_copy_client.cc:523] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.417830 10414 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:13.424485 10401 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.424702 10401 tablet_bootstrap.cc:492] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:13.424855 10401 ts_tablet_manager.cc:1367] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.000s
I20240503 07:59:13.425246 10401 raft_consensus.cc:348] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.425347 10401 raft_consensus.cc:729] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:13.425429 10401 consensus_queue.cc:260] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.425721 10401 ts_tablet_manager.cc:1398] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.426270 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-824c42be8152494fad2cbc76ece56981 received from {username='slave'} at 127.0.99.135:33533
I20240503 07:59:13.426331 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-824c42be8152494fad2cbc76ece56981 on tablet 824c42be8152494fad2cbc76ece56981 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:13.435914 8977 consensus_queue.cc:237] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:13.437250 8575 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.437304 6296 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:13.437659 8912 consensus_peers.cc:485] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94d5216ff8654561b44a0cfd0f9b1625. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.437718 9989 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.437851 10502 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.439373 9989 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:13.439734 6296 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:13.441423 8575 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:13.441919 6157 catalog_manager.cc:5484] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.445904 10414 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.446132 10414 tablet_bootstrap.cc:492] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:13.446277 10414 ts_tablet_manager.cc:1367] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.003s
I20240503 07:59:13.446676 10414 raft_consensus.cc:348] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.446772 10414 raft_consensus.cc:729] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:13.446858 10414 consensus_queue.cc:260] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.447136 10414 ts_tablet_manager.cc:1398] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.447773 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-a2762b2e6de842bb8c898aea5b3910ba received from {username='slave'} at 127.0.99.136:36575
I20240503 07:59:13.447855 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-a2762b2e6de842bb8c898aea5b3910ba on tablet a2762b2e6de842bb8c898aea5b3910ba with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:13.453042 8712 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:13.454373 8575 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:13.454603 8645 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f534eefb7c334889a9c5d02175ae2d94. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.454646 8977 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.454790 9189 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.454988 9400 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:13.456561 9189 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:13.457155 8977 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:13.457252 8575 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:13.459508 6153 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.470153 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:13.476356 8560 tablet_service.cc:1467] Processing CreateTablet for tablet b61209b73f1f4872b5eccc823b1fa3df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.476611 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.477061 8561 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4c776eac24f739d85e09707cd751b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.477284 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4c776eac24f739d85e09707cd751b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.477236 8822 tablet_service.cc:1467] Processing CreateTablet for tablet b61209b73f1f4872b5eccc823b1fa3df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.477464 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.477420 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 873c7786052b4b9c97cc0c1bcb6e7a03 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.477628 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c7786052b4b9c97cc0c1bcb6e7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.479200 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 873c7786052b4b9c97cc0c1bcb6e7a03 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.479249 6279 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4c776eac24f739d85e09707cd751b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.479447 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c7786052b4b9c97cc0c1bcb6e7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.479485 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4c776eac24f739d85e09707cd751b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.479652 8692 tablet_service.cc:1467] Processing CreateTablet for tablet b61209b73f1f4872b5eccc823b1fa3df (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.479871 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.480929 10564 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.481081 10565 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:13.481869 10564 tablet_bootstrap.cc:654] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.481902 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 873c7786052b4b9c97cc0c1bcb6e7a03 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.482117 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873c7786052b4b9c97cc0c1bcb6e7a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.482206 10565 tablet_bootstrap.cc:654] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.482591 10607 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:13.482740 10567 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:13.483140 10564 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:13.483208 10564 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.483394 10607 tablet_bootstrap.cc:654] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.483505 10567 tablet_bootstrap.cc:654] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.483649 10564 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.483740 10564 raft_consensus.cc:374] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.483786 10564 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.483919 10564 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.483978 8690 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4c776eac24f739d85e09707cd751b (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:13.484175 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4c776eac24f739d85e09707cd751b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.484215 10564 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.484273 10607 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:13.484354 10607 ts_tablet_manager.cc:1367] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.484393 10564 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.484707 10607 raft_consensus.cc:348] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.484830 10607 raft_consensus.cc:374] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.484869 10607 raft_consensus.cc:729] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.484984 10607 consensus_queue.cc:260] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.485222 10564 tablet_bootstrap.cc:654] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.485268 10607 ts_tablet_manager.cc:1398] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.486642 10564 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:13.486719 10564 ts_tablet_manager.cc:1367] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:13.486768 10567 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:13.486850 10567 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:13.487103 10564 raft_consensus.cc:348] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.487188 10564 raft_consensus.cc:374] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.487215 10564 raft_consensus.cc:729] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.487198 10567 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.487279 10567 raft_consensus.cc:374] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.487308 10567 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.487311 10564 consensus_queue.cc:260] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.487404 10567 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.487577 10564 ts_tablet_manager.cc:1398] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:13.487637 10567 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.487747 10567 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:13.487910 10606 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:13.488351 10567 tablet_bootstrap.cc:654] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.489027 10565 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:13.489104 10565 ts_tablet_manager.cc:1367] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:13.489235 10606 tablet_bootstrap.cc:654] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.489528 10565 raft_consensus.cc:348] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.489620 10565 raft_consensus.cc:374] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.489650 10565 raft_consensus.cc:729] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.489755 10565 consensus_queue.cc:260] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.490000 10565 ts_tablet_manager.cc:1398] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.490116 10606 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:13.490185 10606 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:13.490530 10606 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.490624 10606 raft_consensus.cc:374] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.490653 10606 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.490768 10606 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.491034 10606 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.491176 10606 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:13.491959 10606 tablet_bootstrap.cc:654] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.493285 10567 tablet_bootstrap.cc:492] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:13.493364 10567 ts_tablet_manager.cc:1367] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:13.493755 10567 raft_consensus.cc:348] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.493854 10567 raft_consensus.cc:374] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.493883 10567 raft_consensus.cc:729] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.493994 10567 consensus_queue.cc:260] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.494149 8712 raft_consensus.cc:1203] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.494186 10567 ts_tablet_manager.cc:1398] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:13.494313 10567 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:13.494992 10606 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:13.495038 10567 tablet_bootstrap.cc:654] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:13.495332 10606 ts_tablet_manager.cc:1367] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:13.495945 10606 raft_consensus.cc:348] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.496037 10606 raft_consensus.cc:374] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.496068 10606 raft_consensus.cc:729] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.496192 10606 consensus_queue.cc:260] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.497159 10606 ts_tablet_manager.cc:1398] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.498201 10567 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:13.498275 10567 ts_tablet_manager.cc:1367] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:13.498626 10567 raft_consensus.cc:348] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.498716 10567 raft_consensus.cc:374] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:13.498744 10567 raft_consensus.cc:729] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:13.498857 10567 consensus_queue.cc:260] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.499114 10567 ts_tablet_manager.cc:1398] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.503491 10412 raft_consensus.cc:482] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.503576 10412 raft_consensus.cc:504] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.504030 10412 leader_election.cc:288] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:13.506752 9400 raft_consensus.cc:482] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.506835 9400 raft_consensus.cc:504] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.507161 9400 leader_election.cc:288] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.508057 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de1f1d3f05d42b39563d6cb5857f7c3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:13.508191 8977 raft_consensus.cc:2454] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.508698 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de1f1d3f05d42b39563d6cb5857f7c3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:13.508852 8843 raft_consensus.cc:2454] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.509058 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4c776eac24f739d85e09707cd751b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:13.509153 8647 leader_election.cc:302] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:13.509186 6296 raft_consensus.cc:2454] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:13.509292 9400 raft_consensus.cc:2781] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.509330 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4c776eac24f739d85e09707cd751b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:13.509420 8712 raft_consensus.cc:2454] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:13.509474 8514 leader_election.cc:302] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:13.509624 10412 raft_consensus.cc:2781] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.509667 10412 raft_consensus.cc:482] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.509713 10412 raft_consensus.cc:3037] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.509336 9400 raft_consensus.cc:482] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.509979 9400 raft_consensus.cc:3037] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.510650 10412 raft_consensus.cc:504] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.510723 9400 raft_consensus.cc:504] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.510968 10412 leader_election.cc:288] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:13.511036 9400 leader_election.cc:288] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.511577 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4c776eac24f739d85e09707cd751b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:13.511682 6296 raft_consensus.cc:3037] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.511710 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de1f1d3f05d42b39563d6cb5857f7c3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:13.511729 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4c776eac24f739d85e09707cd751b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:13.511801 8843 raft_consensus.cc:3037] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.511808 8712 raft_consensus.cc:3037] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.512545 8843 raft_consensus.cc:2454] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.512549 6296 raft_consensus.cc:2454] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:13.512549 8712 raft_consensus.cc:2454] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:13.512871 8514 leader_election.cc:302] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:13.512892 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de1f1d3f05d42b39563d6cb5857f7c3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:13.512992 8977 raft_consensus.cc:3037] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.513051 10412 raft_consensus.cc:2781] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.513121 10412 raft_consensus.cc:686] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:13.513231 10412 consensus_queue.cc:237] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:13.513748 8977 raft_consensus.cc:2454] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.514006 8647 leader_election.cc:302] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:13.514153 9400 raft_consensus.cc:2781] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.514226 9400 raft_consensus.cc:686] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:13.514328 9400 consensus_queue.cc:237] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.515187 6152 catalog_manager.cc:5484] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.516287 6153 catalog_manager.cc:5484] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.521905 10079 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.521984 10079 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.522276 10079 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.522588 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:13.522635 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:13.522696 8977 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.522758 8843 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.523255 8647 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:13.523406 10079 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.523453 10079 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.523487 10079 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.524147 10079 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.524425 10079 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.524673 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:13.524796 8843 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.524749 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:13.524875 8977 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.525596 8843 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.525677 8977 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.525840 8647 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:13.525982 10079 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.526050 10079 raft_consensus.cc:686] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:13.526155 10079 consensus_queue.cc:237] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:13.527632 6153 catalog_manager.cc:5484] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.538736 8843 raft_consensus.cc:1203] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.554227 8575 raft_consensus.cc:1203] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.593335 9278 raft_consensus.cc:1050] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:13.593641 9278 consensus_queue.cc:237] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:13.594902 6691 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.594967 8575 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.595005 6296 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.595384 9278 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.595558 9963 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.595696 8893 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.597457 9278 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.597823 6296 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.599720 8575 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.600119 6153 catalog_manager.cc:5484] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.601660 6690 consensus_queue.cc:237] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
W20240503 07:59:13.603019 6627 consensus_peers.cc:485] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e0f5369e8349eba7188b39ae6bac30. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.603080 6296 raft_consensus.cc:1261] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.603585 6428 raft_consensus.cc:1261] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.603770 10254 consensus_queue.cc:1035] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.604835 10254 consensus_queue.cc:1035] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.606537 6691 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.607367 10254 raft_consensus.cc:2932] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.608422 6558 consensus_queue.cc:237] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:13.609581 8575 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:13.610049 6298 raft_consensus.cc:1261] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:13.610417 8893 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.610657 8893 consensus_queue.cc:1035] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.612141 9277 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.614044 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d855e5edf05471983dfdbd40f37560f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:13.614487 8575 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.614970 6298 raft_consensus.cc:2932] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.616499 6296 raft_consensus.cc:2932] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.617450 6152 catalog_manager.cc:5484] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.617741 6153 catalog_manager.cc:5484] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.618849 6428 raft_consensus.cc:2932] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.620728 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d855e5edf05471983dfdbd40f37560f with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:13.621038 9175 tablet_replica.cc:330] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:13.621158 9175 raft_consensus.cc:2229] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.621300 9175 raft_consensus.cc:2258] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.621883 9175 ts_tablet_manager.cc:1875] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:13.623679 9175 ts_tablet_manager.cc:1888] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:13.623726 9175 log.cc:1196] T 7d855e5edf05471983dfdbd40f37560f P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:13.624105 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 7d855e5edf05471983dfdbd40f37560f (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:13.630002 8976 raft_consensus.cc:1203] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.639225 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:13.643164 8843 raft_consensus.cc:1261] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.643179 8976 raft_consensus.cc:1261] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:13.643595 10079 consensus_queue.cc:1035] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.643837 9400 consensus_queue.cc:1035] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.646862 6153 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]). Expected version 1 got 0
I20240503 07:59:13.647387 6157 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]). Expected version 1 got 0
I20240503 07:59:13.647701 10549 mvcc.cc:205] Tried to move back new op lower bound from 7023506037318324224 to 7023506036795990016. Current Snapshot: MvccSnapshot[applied={T|T < 7023506037318324224}]
I20240503 07:59:13.648438 6157 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]). Expected version 1 got 0
I20240503 07:59:13.649549 6153 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]). Expected version 1 got 0
I20240503 07:59:13.649783 10727 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.650256 10727 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:13.650611 6153 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98]). Expected version 1 got 0
I20240503 07:59:13.651060 10729 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.651440 10729 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:13.653803 10728 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:13.654189 10728 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:13.654698 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98] alter complete (version 1)
I20240503 07:59:13.663233 6153 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 07:59:13.663349 6153 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 07:59:13.665601 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet 0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:13.666090 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.666121 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.666304 9178 tablet_replica.cc:330] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.666407 9177 tablet_replica.cc:330] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.666442 9178 raft_consensus.cc:2229] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.666401 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 065d5d0851e840939802896f9bddb84c
I20240503 07:59:13.666517 9177 raft_consensus.cc:2229] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.666574 9178 raft_consensus.cc:2258] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.666669 9177 raft_consensus.cc:2258] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.666676 8690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:13.666831 10259 tablet_replica.cc:330] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:13.666975 10259 raft_consensus.cc:2229] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:13.667073 10259 raft_consensus.cc:2258] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:13.667094 9177 ts_tablet_manager.cc:1875] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.667095 9178 ts_tablet_manager.cc:1875] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.667264 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 065d5d0851e840939802896f9bddb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:13.667416 10522 tablet_replica.cc:330] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:13.667531 10259 ts_tablet_manager.cc:1875] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.667552 10522 raft_consensus.cc:2229] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.667677 10522 raft_consensus.cc:2258] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.668180 10522 ts_tablet_manager.cc:1875] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.669164 9178 ts_tablet_manager.cc:1888] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.669170 9177 ts_tablet_manager.cc:1888] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.669226 9178 log.cc:1196] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:13.669226 9177 log.cc:1196] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:13.669211 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 065d5d0851e840939802896f9bddb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:13.669476 9177 ts_tablet_manager.cc:1909] T 0e3f58a27f2a4a2593e8219b1d7bd894 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.669476 9178 ts_tablet_manager.cc:1909] T 0e3f58a27f2a4a2593e8219b1d7bd894 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:13.669696 10414 ts_tablet_manager.cc:902] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:13.670161 10414 tablet_copy_client.cc:308] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:13.670158 10522 ts_tablet_manager.cc:1888] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:13.670221 10522 log.cc:1196] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/065d5d0851e840939802896f9bddb84c
I20240503 07:59:13.670207 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f58a27f2a4a2593e8219b1d7bd894 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:13.670384 9176 tablet_replica.cc:330] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:13.670465 10522 ts_tablet_manager.cc:1909] T 065d5d0851e840939802896f9bddb84c P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:13.670512 9176 raft_consensus.cc:2229] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.670769 9176 raft_consensus.cc:2258] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.671049 9175 tablet_replica.cc:330] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:13.671170 9175 raft_consensus.cc:2229] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.671245 9176 ts_tablet_manager.cc:1875] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.671350 9175 raft_consensus.cc:2258] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.671542 10259 ts_tablet_manager.cc:1888] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.671600 10259 log.cc:1196] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:13.671839 10259 ts_tablet_manager.cc:1909] T 0e3f58a27f2a4a2593e8219b1d7bd894 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:13.671826 9175 ts_tablet_manager.cc:1875] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.672294 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.673507 9176 ts_tablet_manager.cc:1888] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:13.673561 9176 log.cc:1196] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e3f58a27f2a4a2593e8219b1d7bd894
I20240503 07:59:13.673581 9175 ts_tablet_manager.cc:1888] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:13.673627 9175 log.cc:1196] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/065d5d0851e840939802896f9bddb84c
I20240503 07:59:13.673746 9176 ts_tablet_manager.cc:1909] T 0e3f58a27f2a4a2593e8219b1d7bd894 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:13.673786 9175 ts_tablet_manager.cc:1909] T 065d5d0851e840939802896f9bddb84c P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:13.674485 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.674544 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.674664 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 8861b52c306e43d8b0615202cf7d8e35 from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:36575)
I20240503 07:59:13.674755 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 8861b52c306e43d8b0615202cf7d8e35 from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:36575: session id = 3d08a06d53b14381bee6223c4e1a6027-8861b52c306e43d8b0615202cf7d8e35
I20240503 07:59:13.675592 6712 tablet_copy_source_session.cc:205] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.675951 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 065d5d0851e840939802896f9bddb84c (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.676167 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 0e3f58a27f2a4a2593e8219b1d7bd894 (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.676223 6558 consensus_queue.cc:237] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:13.676339 10414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8861b52c306e43d8b0615202cf7d8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.676342 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 065d5d0851e840939802896f9bddb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.676991 9177 tablet_replica.cc:330] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.677170 9177 raft_consensus.cc:2229] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.677367 9177 raft_consensus.cc:2258] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.677536 8575 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:13.677842 9177 ts_tablet_manager.cc:1875] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.677940 9963 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.678280 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 065d5d0851e840939802896f9bddb84c (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.678318 10414 tablet_copy_client.cc:783] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.678432 10414 tablet_copy_client.cc:647] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.678853 6296 raft_consensus.cc:1261] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:13.679050 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet aa754582dc9e4518965f3a2c1836cbee
I20240503 07:59:13.679348 9963 consensus_queue.cc:1035] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.679684 9963 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.679919 10414 tablet_copy_client.cc:523] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.680138 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet aa754582dc9e4518965f3a2c1836cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.680275 9178 tablet_replica.cc:330] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.680306 9177 ts_tablet_manager.cc:1888] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:13.680380 9177 log.cc:1196] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/065d5d0851e840939802896f9bddb84c
I20240503 07:59:13.680385 9178 raft_consensus.cc:2229] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.680500 9178 raft_consensus.cc:2258] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.680639 9177 ts_tablet_manager.cc:1909] T 065d5d0851e840939802896f9bddb84c P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.680944 9178 ts_tablet_manager.cc:1875] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.681025 10414 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:13.681429 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7e3555fb9214a40bd8dbf937b87cbaa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:13.681594 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet aa754582dc9e4518965f3a2c1836cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:13.681754 9176 tablet_replica.cc:330] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:13.681869 9176 raft_consensus.cc:2229] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.682049 9176 raft_consensus.cc:2258] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.682734 9176 ts_tablet_manager.cc:1875] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.682791 9178 ts_tablet_manager.cc:1888] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.682857 9178 log.cc:1196] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa754582dc9e4518965f3a2c1836cbee
I20240503 07:59:13.683094 9178 ts_tablet_manager.cc:1909] T aa754582dc9e4518965f3a2c1836cbee P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:13.683694 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet aa754582dc9e4518965f3a2c1836cbee (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.683992 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 065d5d0851e840939802896f9bddb84c (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.684162 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet aa754582dc9e4518965f3a2c1836cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:13.684651 6296 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.684908 8575 raft_consensus.cc:2932] T c7e3555fb9214a40bd8dbf937b87cbaa P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:13.685173 9176 ts_tablet_manager.cc:1888] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.685235 9176 log.cc:1196] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa754582dc9e4518965f3a2c1836cbee
I20240503 07:59:13.685501 9176 ts_tablet_manager.cc:1909] T aa754582dc9e4518965f3a2c1836cbee P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:13.685454 6157 catalog_manager.cc:5484] T c7e3555fb9214a40bd8dbf937b87cbaa P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 7, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:13.686136 9177 tablet_replica.cc:330] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:13.686267 9177 raft_consensus.cc:2229] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.686393 9177 raft_consensus.cc:2258] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.686836 9177 ts_tablet_manager.cc:1875] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:13.688180 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet aa754582dc9e4518965f3a2c1836cbee (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.689034 9177 ts_tablet_manager.cc:1888] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:13.689980 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e3555fb9214a40bd8dbf937b87cbaa with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:13.690177 9178 tablet_replica.cc:330] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:13.690304 9178 raft_consensus.cc:2229] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.690429 9178 raft_consensus.cc:2258] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.691210 9178 ts_tablet_manager.cc:1875] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:13.692994 9177 log.cc:1196] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa754582dc9e4518965f3a2c1836cbee
I20240503 07:59:13.693405 9177 ts_tablet_manager.cc:1909] T aa754582dc9e4518965f3a2c1836cbee P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:13.694123 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet aa754582dc9e4518965f3a2c1836cbee (table rebalance_test_table_14 [id=18098099334d40cead69112e658daaef]) successfully deleted
I20240503 07:59:13.695075 9178 ts_tablet_manager.cc:1888] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 07:59:13.695151 9178 log.cc:1196] T c7e3555fb9214a40bd8dbf937b87cbaa P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7e3555fb9214a40bd8dbf937b87cbaa
I20240503 07:59:13.695590 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet c7e3555fb9214a40bd8dbf937b87cbaa (table rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]) successfully deleted
I20240503 07:59:13.706789 10414 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.707043 10414 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:13.707204 10414 ts_tablet_manager.cc:1367] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.004s
I20240503 07:59:13.707589 10414 raft_consensus.cc:348] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.707748 10414 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:13.707868 10414 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:13.708158 10414 ts_tablet_manager.cc:1398] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:13.708631 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-8861b52c306e43d8b0615202cf7d8e35 received from {username='slave'} at 127.0.99.136:36575
I20240503 07:59:13.708715 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-8861b52c306e43d8b0615202cf7d8e35 on tablet 8861b52c306e43d8b0615202cf7d8e35 with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:13.755090 9288 raft_consensus.cc:1050] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:13.755492 9288 consensus_queue.cc:237] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:13.756978 8843 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.757210 6558 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.757367 6296 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.757700 10801 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.758637 9952 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.758997 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:13.759824 8895 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.761186 8895 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.762202 10810 ts_tablet_manager.cc:902] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:13.762547 8843 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.763338 10810 tablet_copy_client.cc:308] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:13.763199 6296 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.764601 6558 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.765394 6157 catalog_manager.cc:5484] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.766467 9400 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:13.766561 9400 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.766952 9400 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:13.767304 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:13.767321 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:13.767431 8843 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.767431 8575 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:13.767695 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 2f943ecfe59f432c861a54b16c62572f from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:52241)
I20240503 07:59:13.767722 8645 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:13.767789 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 2f943ecfe59f432c861a54b16c62572f from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:52241: session id = c8d71787690245e681c85a087086e7ed-2f943ecfe59f432c861a54b16c62572f
I20240503 07:59:13.768010 9400 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:13.768078 9400 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:13.768126 9400 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.768545 6318 tablet_copy_source_session.cc:205] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.768829 9400 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.769210 9400 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:13.769408 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f943ecfe59f432c861a54b16c62572f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.769562 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:13.769673 8575 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.770565 8575 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.770994 8645 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:13.771175 9400 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:13.771262 9400 raft_consensus.cc:686] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:13.771396 9400 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:13.771996 10810 tablet_copy_client.cc:783] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.772123 10810 tablet_copy_client.cc:647] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.773317 10401 ts_tablet_manager.cc:902] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.773655 6150 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:13.773864 10401 tablet_copy_client.cc:308] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41089
I20240503 07:59:13.774446 8997 tablet_copy_service.cc:140] P 3d08a06d53b14381bee6223c4e1a6027: Received BeginTabletCopySession request for tablet a01fa99be75f498a8394aabbfabb3f5a from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:40449)
I20240503 07:59:13.774569 8997 tablet_copy_service.cc:161] P 3d08a06d53b14381bee6223c4e1a6027: Beginning new tablet copy session on tablet a01fa99be75f498a8394aabbfabb3f5a from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:40449: session id = 616fdda6ba714cb7be01a564de2b9db2-a01fa99be75f498a8394aabbfabb3f5a
I20240503 07:59:13.775704 10727 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.775842 8997 tablet_copy_source_session.cc:205] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.775990 10727 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:13.776413 10728 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.776669 10728 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:13.776940 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa99be75f498a8394aabbfabb3f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.778082 6428 consensus_queue.cc:237] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:13.779614 8842 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:13.779927 6296 raft_consensus.cc:1261] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:13.779033 10401 tablet_copy_client.cc:783] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.780105 10401 tablet_copy_client.cc:647] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.780555 9288 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.780735 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=fb763ed6050840928d85eece5bf5ec98] alter complete (version 2)
I20240503 07:59:13.780925 9288 consensus_queue.cc:1035] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:13.782159 10401 tablet_copy_client.cc:523] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.782593 9952 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.782936 10729 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:13.783188 10729 tablet.cc:1620] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:13.783227 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:13.783305 8843 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:13.783346 10401 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:13.784130 8843 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:13.784720 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5604af852814da58385f072eb9600aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:13.785342 8842 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.785396 6296 raft_consensus.cc:2932] T f5604af852814da58385f072eb9600aa P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:13.785696 6156 catalog_manager.cc:5484] T f5604af852814da58385f072eb9600aa P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:13.788384 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet f5604af852814da58385f072eb9600aa with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:13.788594 9176 tablet_replica.cc:330] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:13.788722 9176 raft_consensus.cc:2229] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.788897 9176 raft_consensus.cc:2258] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.789496 9176 ts_tablet_manager.cc:1875] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:13.790704 10401 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.790894 10401 tablet_bootstrap.cc:492] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:13.791028 10401 ts_tablet_manager.cc:1367] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:13.791371 9176 ts_tablet_manager.cc:1888] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:13.791435 9176 log.cc:1196] T f5604af852814da58385f072eb9600aa P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f5604af852814da58385f072eb9600aa
I20240503 07:59:13.791391 10401 raft_consensus.cc:348] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.791496 10401 raft_consensus.cc:729] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:13.791580 10401 consensus_queue.cc:260] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.791814 10401 ts_tablet_manager.cc:1398] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:13.792204 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet f5604af852814da58385f072eb9600aa (table rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]) successfully deleted
I20240503 07:59:13.792253 8997 tablet_copy_service.cc:342] P 3d08a06d53b14381bee6223c4e1a6027: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-a01fa99be75f498a8394aabbfabb3f5a received from {username='slave'} at 127.0.99.135:40449
I20240503 07:59:13.792316 8997 tablet_copy_service.cc:433] P 3d08a06d53b14381bee6223c4e1a6027: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-a01fa99be75f498a8394aabbfabb3f5a on tablet a01fa99be75f498a8394aabbfabb3f5a with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:13.792598 10810 tablet_copy_client.cc:523] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.794255 10810 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:13.794885 6428 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.796557 8977 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:13.796675 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:13.796765 6363 consensus_peers.cc:485] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5631f3e97fe04013a2a7140f44260003. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.796972 8895 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.797217 10801 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.798934 8895 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.799328 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.799525 8977 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.801312 6150 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.810462 6430 consensus_queue.cc:237] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
W20240503 07:59:13.812103 6363 consensus_peers.cc:485] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60f643dce3eb4c669b36b0714a31651b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.812458 6558 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.812557 6691 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:13.812866 9952 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.813040 9288 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:13.814894 9288 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.816483 6691 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.816815 6558 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:13.817931 6157 catalog_manager.cc:5484] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:13.828060 10810 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.828373 10810 tablet_bootstrap.cc:492] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:13.828579 10810 ts_tablet_manager.cc:1367] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.000s
I20240503 07:59:13.829352 10810 raft_consensus.cc:348] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.829486 10810 raft_consensus.cc:729] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:13.829597 10810 consensus_queue.cc:260] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:13.830461 10810 ts_tablet_manager.cc:1398] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:13.830880 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-2f943ecfe59f432c861a54b16c62572f received from {username='slave'} at 127.0.99.133:52241
I20240503 07:59:13.830953 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session c8d71787690245e681c85a087086e7ed-2f943ecfe59f432c861a54b16c62572f on tablet 2f943ecfe59f432c861a54b16c62572f with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:13.835350 10844 ts_tablet_manager.cc:902] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:13.836684 10844 tablet_copy_client.cc:308] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:13.837370 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet f534eefb7c334889a9c5d02175ae2d94 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:38091)
I20240503 07:59:13.837492 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet f534eefb7c334889a9c5d02175ae2d94 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:38091: session id = ad910f92ef9d4db8a8962cecba289f8a-f534eefb7c334889a9c5d02175ae2d94
I20240503 07:59:13.838655 8732 tablet_copy_source_session.cc:205] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.839923 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f534eefb7c334889a9c5d02175ae2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.842054 10844 tablet_copy_client.cc:783] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.842393 10844 tablet_copy_client.cc:647] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.844466 10844 tablet_copy_client.cc:523] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.846181 10844 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:13.850137 10844 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.850423 10844 tablet_bootstrap.cc:492] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:13.850565 10844 ts_tablet_manager.cc:1367] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 07:59:13.850939 10844 raft_consensus.cc:348] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:13.851049 10844 raft_consensus.cc:729] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:13.851138 10844 consensus_queue.cc:260] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:13.851917 10844 ts_tablet_manager.cc:1398] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:13.852694 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-f534eefb7c334889a9c5d02175ae2d94 received from {username='slave'} at 127.0.99.131:38091
I20240503 07:59:13.853108 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-f534eefb7c334889a9c5d02175ae2d94 on tablet f534eefb7c334889a9c5d02175ae2d94 with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:13.853593 10864 ts_tablet_manager.cc:902] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:13.854279 10864 tablet_copy_client.cc:308] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41089
I20240503 07:59:13.854830 8997 tablet_copy_service.cc:140] P 3d08a06d53b14381bee6223c4e1a6027: Received BeginTabletCopySession request for tablet 94d5216ff8654561b44a0cfd0f9b1625 from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:53975)
I20240503 07:59:13.854931 8997 tablet_copy_service.cc:161] P 3d08a06d53b14381bee6223c4e1a6027: Beginning new tablet copy session on tablet 94d5216ff8654561b44a0cfd0f9b1625 from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:53975: session id = a97073f89fd44702bccb74a9b76f858d-94d5216ff8654561b44a0cfd0f9b1625
I20240503 07:59:13.856321 8997 tablet_copy_source_session.cc:205] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:13.857228 10864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d5216ff8654561b44a0cfd0f9b1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:13.859131 10864 tablet_copy_client.cc:783] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:13.859295 10864 tablet_copy_client.cc:647] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:13.860563 10864 tablet_copy_client.cc:523] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:13.861939 10864 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:13.865442 10864 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:13.865633 10864 tablet_bootstrap.cc:492] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:13.865742 10864 ts_tablet_manager.cc:1367] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 07:59:13.866024 10864 raft_consensus.cc:348] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:13.866147 10864 raft_consensus.cc:729] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:13.866250 10864 consensus_queue.cc:260] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:13.866478 10864 ts_tablet_manager.cc:1398] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:13.867259 8997 tablet_copy_service.cc:342] P 3d08a06d53b14381bee6223c4e1a6027: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-94d5216ff8654561b44a0cfd0f9b1625 received from {username='slave'} at 127.0.99.132:53975
I20240503 07:59:13.867322 8997 tablet_copy_service.cc:433] P 3d08a06d53b14381bee6223c4e1a6027: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-94d5216ff8654561b44a0cfd0f9b1625 on tablet 94d5216ff8654561b44a0cfd0f9b1625 with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:13.883117 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:13.888289 10727 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.888913 10728 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.891295 10729 tablet.cc:1722] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:13.893180 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=fb763ed6050840928d85eece5bf5ec98] alter complete (version 3)
I20240503 07:59:13.909611 8842 raft_consensus.cc:1203] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.928910 6691 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:13.930177 8575 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:13.930251 6558 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:13.930261 6627 consensus_peers.cc:485] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39b01318dc1543cba2cc991e9db91c28. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:13.930584 9347 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.930778 8077 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:13.932437 8296 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.932946 6558 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.933030 8575 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.935317 6157 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:13.936225 10815 consensus_queue.cc:1035] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.939011 10412 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.941772 9400 consensus_queue.cc:1035] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.947299 10412 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:13.990936 8977 raft_consensus.cc:1203] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:13.992744 9952 raft_consensus.cc:1050] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:13.993177 9952 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:13.994580 6558 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.995112 6690 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.995280 8977 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.995529 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.995644 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:13.995906 9952 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.996276 9952 raft_consensus.cc:1050] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:13.996616 9952 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:13.997529 8895 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.997753 6558 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.997969 10801 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.998073 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 1de1f1d3f05d42b39563d6cb5857f7c3
I20240503 07:59:13.998209 9288 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:13.998651 8690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:13.998788 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:13.998843 8712 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:13.998912 10259 tablet_replica.cc:330] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:13.999025 10259 raft_consensus.cc:2229] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:13.999059 10995 tablet_replica.cc:330] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:13.999059 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de1f1d3f05d42b39563d6cb5857f7c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:13 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:13.999197 10259 raft_consensus.cc:2258] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.999212 10522 tablet_replica.cc:330] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:13.999208 10995 raft_consensus.cc:2229] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.999310 10995 raft_consensus.cc:2258] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.999315 10522 raft_consensus.cc:2229] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:13.999415 10522 raft_consensus.cc:2258] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:13.999958 6691 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.000005 10995 ts_tablet_manager.cc:1875] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.000011 10259 ts_tablet_manager.cc:1875] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.000396 10522 ts_tablet_manager.cc:1875] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.002072 10995 ts_tablet_manager.cc:1888] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.002084 10259 ts_tablet_manager.cc:1888] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.002132 10995 log.cc:1196] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1de1f1d3f05d42b39563d6cb5857f7c3
I20240503 07:59:14.002146 10259 log.cc:1196] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1de1f1d3f05d42b39563d6cb5857f7c3
I20240503 07:59:14.002319 10522 ts_tablet_manager.cc:1888] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.002373 10259 ts_tablet_manager.cc:1909] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:14.002377 10522 log.cc:1196] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/1de1f1d3f05d42b39563d6cb5857f7c3
I20240503 07:59:14.002398 10995 ts_tablet_manager.cc:1909] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:14.002583 10522 ts_tablet_manager.cc:1909] T 1de1f1d3f05d42b39563d6cb5857f7c3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:14.003069 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table_new_name [id=fb763ed6050840928d85eece5bf5ec98]) successfully deleted
I20240503 07:59:14.004017 10801 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.004567 9288 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.004848 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table_new_name [id=fb763ed6050840928d85eece5bf5ec98]) successfully deleted
I20240503 07:59:14.006808 8895 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.006913 10801 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.008251 8712 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.008603 8977 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.009649 6156 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.010862 6691 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.011570 6150 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.013763 8296 raft_consensus.cc:1050] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:14.014108 8296 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:14.014668 6430 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:14.015298 6558 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.015405 8575 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.015573 6559 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.016068 6296 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.016595 8077 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.016850 9347 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.016990 8077 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.017555 6560 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.019300 8712 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.020120 9288 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.021445 6430 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.022020 6691 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.023538 8977 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.023732 6558 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.024199 8895 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.024507 6560 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.024518 8895 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.025162 10801 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.026026 10993 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.026335 6560 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.027698 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c01e3ce57874e24ae75a41243fdeaaf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.028998 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 1de1f1d3f05d42b39563d6cb5857f7c3 (table rebalancer_dynamic_table_new_name [id=fb763ed6050840928d85eece5bf5ec98]) successfully deleted
I20240503 07:59:14.029453 6152 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.030848 9288 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.030828 8977 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.032320 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c01e3ce57874e24ae75a41243fdeaaf with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.032509 9175 tablet_replica.cc:330] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.032642 9175 raft_consensus.cc:2229] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.032791 9175 raft_consensus.cc:2258] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.033432 9175 ts_tablet_manager.cc:1875] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.034020 8712 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.034623 6152 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.035460 9175 ts_tablet_manager.cc:1888] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.035522 9175 log.cc:1196] T 8c01e3ce57874e24ae75a41243fdeaaf P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8c01e3ce57874e24ae75a41243fdeaaf
I20240503 07:59:14.035519 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edf2c780d0d64db79dca88eac77d4478 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.035939 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 8c01e3ce57874e24ae75a41243fdeaaf (table rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]) successfully deleted
I20240503 07:59:14.036924 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet edf2c780d0d64db79dca88eac77d4478 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.037228 9175 tablet_replica.cc:330] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.037379 9347 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.037729 6296 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.040102 9175 raft_consensus.cc:2229] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.040254 9175 raft_consensus.cc:2258] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.040216 6560 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.040277 6150 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.040913 6559 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:14.044742 8575 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.046142 9175 ts_tablet_manager.cc:1875] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.048851 6691 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:14.049297 9175 ts_tablet_manager.cc:1888] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.049355 9175 log.cc:1196] T edf2c780d0d64db79dca88eac77d4478 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/edf2c780d0d64db79dca88eac77d4478
I20240503 07:59:14.050307 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet edf2c780d0d64db79dca88eac77d4478 (table rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]) successfully deleted
I20240503 07:59:14.050732 6560 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.051203 8077 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.051254 8575 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.051671 8077 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.053169 8296 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.054034 6560 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.054327 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9772b487ca864ad98e9d79d8435c34c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.055728 6152 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.056068 8575 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.059119 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9772b487ca864ad98e9d79d8435c34c6 with delete_type TABLET_DATA_TOMBSTONED (TS c167a0a433d34b74affed13cbf96e302 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.059450 9177 tablet_replica.cc:330] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.059581 9177 raft_consensus.cc:2229] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.059711 9177 raft_consensus.cc:2258] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.060329 9177 ts_tablet_manager.cc:1875] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.062336 9177 ts_tablet_manager.cc:1888] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.062398 9177 log.cc:1196] T 9772b487ca864ad98e9d79d8435c34c6 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9772b487ca864ad98e9d79d8435c34c6
I20240503 07:59:14.062911 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 9772b487ca864ad98e9d79d8435c34c6 (table rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]) successfully deleted
I20240503 07:59:14.069356 8977 raft_consensus.cc:1203] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:14.077929 9968 raft_consensus.cc:1050] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:14.078281 9968 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.079453 8842 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.079686 6430 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.080042 6560 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.080029 10254 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.080367 10254 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.080821 9347 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.082522 9347 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.084168 6430 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.084378 6560 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.085206 8842 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.085700 6152 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.089184 6691 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.090379 6430 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.090449 8842 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.090704 9968 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.090889 10254 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.092658 8296 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.093089 6430 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.093346 6296 consensus_queue.cc:237] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:14.094159 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 435f73fb9f384b10b353df19037bcac3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.094944 8842 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.095176 6430 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.095553 9803 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:14.095741 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfd4259777e5492c904e6d65a36d92ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.096225 6691 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.096565 11046 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.096551 6157 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.098227 10239 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:14.099403 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 435f73fb9f384b10b353df19037bcac3 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.099633 9176 tablet_replica.cc:330] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.099788 9176 raft_consensus.cc:2229] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.099750 6430 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:14.099951 9176 raft_consensus.cc:2258] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.100697 9176 ts_tablet_manager.cc:1875] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.100927 6691 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:14.101137 6156 catalog_manager.cc:5484] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.103000 9176 ts_tablet_manager.cc:1888] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.103065 9176 log.cc:1196] T 435f73fb9f384b10b353df19037bcac3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/435f73fb9f384b10b353df19037bcac3
I20240503 07:59:14.103979 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:14.104841 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 435f73fb9f384b10b353df19037bcac3 (table rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]) successfully deleted
I20240503 07:59:14.108474 8690 tablet_service.cc:1467] Processing CreateTablet for tablet 62555c2998e14f6e85896ab61f40e2dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]), partition=
I20240503 07:59:14.108589 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 62555c2998e14f6e85896ab61f40e2dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]), partition=
I20240503 07:59:14.108712 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62555c2998e14f6e85896ab61f40e2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.108824 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62555c2998e14f6e85896ab61f40e2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.109050 8561 tablet_service.cc:1467] Processing CreateTablet for tablet 62555c2998e14f6e85896ab61f40e2dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]), partition=
I20240503 07:59:14.109273 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62555c2998e14f6e85896ab61f40e2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.110527 11054 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.110842 11055 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:14.111308 11054 tablet_bootstrap.cc:654] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.111694 11055 tablet_bootstrap.cc:654] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.112551 11056 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.112890 11055 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:14.112977 11055 ts_tablet_manager.cc:1367] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:14.113408 11055 raft_consensus.cc:348] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.113507 11055 raft_consensus.cc:374] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.113538 11055 raft_consensus.cc:729] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.113662 11056 tablet_bootstrap.cc:654] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.113637 11055 consensus_queue.cc:260] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.113740 11054 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:14.113808 11054 ts_tablet_manager.cc:1367] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:14.113902 11055 ts_tablet_manager.cc:1398] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.114234 11054 raft_consensus.cc:348] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.114322 11054 raft_consensus.cc:374] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.114351 11054 raft_consensus.cc:729] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.114454 11054 consensus_queue.cc:260] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.114701 11054 ts_tablet_manager.cc:1398] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.114962 11056 tablet_bootstrap.cc:492] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:14.115319 8712 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.115710 11056 ts_tablet_manager.cc:1367] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:14.116101 11056 raft_consensus.cc:348] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.116197 11056 raft_consensus.cc:374] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.116226 11056 raft_consensus.cc:729] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.116350 11056 consensus_queue.cc:260] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.116622 11056 ts_tablet_manager.cc:1398] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:14.116735 8977 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.116972 6296 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:14.117026 8645 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f635e30fb6974fa1a79d8a46eecc3d5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.117339 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.118046 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.119494 9189 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:14.120234 6296 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:14.120477 8977 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:14.121840 6157 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.142042 10401 ts_tablet_manager.cc:902] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:14.142567 10401 tablet_copy_client.cc:308] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:14.143013 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet b2e0f5369e8349eba7188b39ae6bac30 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:14.143116 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet b2e0f5369e8349eba7188b39ae6bac30 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.144037 6712 tablet_copy_source_session.cc:205] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.145493 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e0f5369e8349eba7188b39ae6bac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.147169 10401 tablet_copy_client.cc:783] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.147292 10401 tablet_copy_client.cc:647] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.148524 10401 tablet_copy_client.cc:523] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.149636 10401 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.153285 10079 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.156174 10079 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.175832 8575 raft_consensus.cc:1203] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:14.176873 10079 raft_consensus.cc:482] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.176954 10079 raft_consensus.cc:504] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.177249 10079 leader_election.cc:288] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:14.177469 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62555c2998e14f6e85896ab61f40e2dc" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:14.177568 8842 raft_consensus.cc:2454] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:14.177917 8647 leader_election.cc:302] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:14.179145 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62555c2998e14f6e85896ab61f40e2dc" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:14.179258 8575 raft_consensus.cc:2454] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:14.179842 10079 raft_consensus.cc:2781] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.179899 10079 raft_consensus.cc:482] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.179939 10079 raft_consensus.cc:3037] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.180840 10079 raft_consensus.cc:504] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.181154 10079 leader_election.cc:288] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:14.181344 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62555c2998e14f6e85896ab61f40e2dc" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:14.181433 8842 raft_consensus.cc:3037] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.181581 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62555c2998e14f6e85896ab61f40e2dc" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:14.181667 8575 raft_consensus.cc:3037] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.182188 8842 raft_consensus.cc:2454] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:14.182449 8647 leader_election.cc:302] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:14.182531 8575 raft_consensus.cc:2454] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:14.182888 10079 raft_consensus.cc:2781] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.182960 10401 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.182982 10079 raft_consensus.cc:686] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:14.183096 10079 consensus_queue.cc:237] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:14.183158 10401 tablet_bootstrap.cc:492] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:14.183276 10401 ts_tablet_manager.cc:1367] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.000s
I20240503 07:59:14.183697 10401 raft_consensus.cc:348] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.183786 10401 raft_consensus.cc:729] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:14.183876 10401 consensus_queue.cc:260] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.184846 6157 catalog_manager.cc:5484] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.185648 10401 ts_tablet_manager.cc:1398] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.185662 9278 raft_consensus.cc:1050] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:14.186059 9278 consensus_queue.cc:237] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.186525 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-b2e0f5369e8349eba7188b39ae6bac30 received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:14.186589 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-b2e0f5369e8349eba7188b39ae6bac30 on tablet b2e0f5369e8349eba7188b39ae6bac30 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:14.188169 6430 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.188304 8977 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.188369 6691 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.188664 9277 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.188974 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 07:59:14.189004 9277 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.189069 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 07:59:14.189131 9963 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.191596 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:14.191817 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 01118afa143f48fabe979e86d0b10c5f
I20240503 07:59:14.192014 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet d804e4a97c7d4238944fd03721b4f180
I20240503 07:59:14.192129 8690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:14.192139 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.192270 10259 tablet_replica.cc:330] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:14.192332 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9ce52b9ae543cca72fb92efafc3c4d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.192375 10259 raft_consensus.cc:2229] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.192415 9178 tablet_replica.cc:330] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.192489 10259 raft_consensus.cc:2258] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.192528 9178 raft_consensus.cc:2229] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.192571 9176 tablet_replica.cc:330] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.192639 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet d804e4a97c7d4238944fd03721b4f180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.192689 9178 raft_consensus.cc:2258] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.192687 9176 raft_consensus.cc:2229] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.192873 9176 raft_consensus.cc:2258] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.192977 10259 ts_tablet_manager.cc:1875] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.193208 9178 ts_tablet_manager.cc:1875] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.193295 9176 ts_tablet_manager.cc:1875] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.193830 9278 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.194852 10259 ts_tablet_manager.cc:1888] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.194924 10259 log.cc:1196] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:14.195153 10259 ts_tablet_manager.cc:1909] T 0e9ce52b9ae543cca72fb92efafc3c4d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:14.195168 9176 ts_tablet_manager.cc:1888] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.195216 9176 log.cc:1196] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:14.195276 9178 ts_tablet_manager.cc:1888] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.195335 9178 log.cc:1196] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e9ce52b9ae543cca72fb92efafc3c4d
I20240503 07:59:14.195441 9176 ts_tablet_manager.cc:1909] T 0e9ce52b9ae543cca72fb92efafc3c4d P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:14.195499 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 01118afa143f48fabe979e86d0b10c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.195588 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet d804e4a97c7d4238944fd03721b4f180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.195719 9177 tablet_replica.cc:330] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.195827 9177 raft_consensus.cc:2229] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.195938 9177 raft_consensus.cc:2258] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.196305 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 01118afa143f48fabe979e86d0b10c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.196378 9177 ts_tablet_manager.cc:1875] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.196421 9178 ts_tablet_manager.cc:1909] T 0e9ce52b9ae543cca72fb92efafc3c4d P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:14.196892 6156 catalog_manager.cc:5484] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.196946 6430 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.197957 9176 tablet_replica.cc:330] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.198063 9176 raft_consensus.cc:2229] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.198105 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 0e9ce52b9ae543cca72fb92efafc3c4d (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.198163 9176 raft_consensus.cc:2258] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.198390 9177 ts_tablet_manager.cc:1888] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.198443 9177 log.cc:1196] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/01118afa143f48fabe979e86d0b10c5f
I20240503 07:59:14.198571 9176 ts_tablet_manager.cc:1875] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.198652 9177 ts_tablet_manager.cc:1909] T 01118afa143f48fabe979e86d0b10c5f P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:14.199200 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 01118afa143f48fabe979e86d0b10c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.199545 9175 tablet_replica.cc:330] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.199625 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet d804e4a97c7d4238944fd03721b4f180 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.199673 9175 raft_consensus.cc:2229] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.199816 9175 raft_consensus.cc:2258] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.199946 9177 tablet_replica.cc:330] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.200074 9177 raft_consensus.cc:2229] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.200147 9176 ts_tablet_manager.cc:1888] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.200191 9177 raft_consensus.cc:2258] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.200209 9176 log.cc:1196] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d804e4a97c7d4238944fd03721b4f180
I20240503 07:59:14.200249 9175 ts_tablet_manager.cc:1875] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.200435 9176 ts_tablet_manager.cc:1909] T d804e4a97c7d4238944fd03721b4f180 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:14.200593 9177 ts_tablet_manager.cc:1875] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.201223 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 0e9ce52b9ae543cca72fb92efafc3c4d (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.201437 9178 tablet_replica.cc:330] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.201599 9178 raft_consensus.cc:2229] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.201756 9178 raft_consensus.cc:2258] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.202018 9177 ts_tablet_manager.cc:1888] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.202082 9177 log.cc:1196] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d804e4a97c7d4238944fd03721b4f180
I20240503 07:59:14.202172 9175 ts_tablet_manager.cc:1888] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.202231 9178 ts_tablet_manager.cc:1875] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.202245 9175 log.cc:1196] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/01118afa143f48fabe979e86d0b10c5f
I20240503 07:59:14.202276 9177 ts_tablet_manager.cc:1909] T d804e4a97c7d4238944fd03721b4f180 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:14.202468 9175 ts_tablet_manager.cc:1909] T 01118afa143f48fabe979e86d0b10c5f P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:14.202965 9175 tablet_replica.cc:330] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.203078 9175 raft_consensus.cc:2229] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.203024 8977 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.203229 9175 raft_consensus.cc:2258] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.203267 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 01118afa143f48fabe979e86d0b10c5f (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.203377 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet d804e4a97c7d4238944fd03721b4f180 (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.203635 9175 ts_tablet_manager.cc:1875] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.203871 6691 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.204093 9178 ts_tablet_manager.cc:1888] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.204149 9178 log.cc:1196] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/01118afa143f48fabe979e86d0b10c5f
I20240503 07:59:14.204270 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 01118afa143f48fabe979e86d0b10c5f (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.204360 9178 ts_tablet_manager.cc:1909] T 01118afa143f48fabe979e86d0b10c5f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:14.204658 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet d804e4a97c7d4238944fd03721b4f180 (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.205852 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 01118afa143f48fabe979e86d0b10c5f (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.206943 6560 consensus_queue.cc:237] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.207034 9175 ts_tablet_manager.cc:1888] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.207089 9175 log.cc:1196] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d804e4a97c7d4238944fd03721b4f180
I20240503 07:59:14.207340 9175 ts_tablet_manager.cc:1909] T d804e4a97c7d4238944fd03721b4f180 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:14.207713 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 0e9ce52b9ae543cca72fb92efafc3c4d (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.207828 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet d804e4a97c7d4238944fd03721b4f180 (table rebalance_test_table_16 [id=e90fb89304104618a7cac10234d099b3]) successfully deleted
I20240503 07:59:14.207944 6691 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.208079 8977 raft_consensus.cc:1261] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.208253 11112 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.208745 11112 consensus_queue.cc:1035] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.210150 9278 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.210983 6691 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.211038 8977 raft_consensus.cc:2932] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.212662 6152 catalog_manager.cc:5484] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.213591 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.214790 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.215021 9178 tablet_replica.cc:330] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.215123 9178 raft_consensus.cc:2229] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.215246 9178 raft_consensus.cc:2258] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.215816 9178 ts_tablet_manager.cc:1875] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.217593 9178 ts_tablet_manager.cc:1888] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.217723 9178 log.cc:1196] T 4b9870eac6ad47d4a9ad5694dcaa1b1f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b9870eac6ad47d4a9ad5694dcaa1b1f
I20240503 07:59:14.218603 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f (table rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]) successfully deleted
I20240503 07:59:14.255909 6691 consensus_queue.cc:237] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.257607 6560 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.257968 6296 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.258352 8077 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.258631 10254 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:14.258877 6627 consensus_peers.cc:485] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d3f03610866454595f4b6b87b7cd9cb. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.260258 8077 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:14.260659 6560 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:14.260659 6296 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:14.263259 6157 catalog_manager.cc:5484] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:14.273905 8842 consensus_queue.cc:237] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:14.275215 8575 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.275241 8977 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:14.275451 8776 consensus_peers.cc:485] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5060df68b95e48a8a849e77ba926e608. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.275617 10095 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.275799 11166 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.277606 10916 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:14.278026 8977 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:14.279353 8575 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:14.280104 6152 catalog_manager.cc:5484] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.289141 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:14.294602 8957 tablet_service.cc:1467] Processing CreateTablet for tablet a833fb4980c04673a335d020a3ad3dd3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.294775 8824 tablet_service.cc:1467] Processing CreateTablet for tablet a833fb4980c04673a335d020a3ad3dd3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.294885 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a833fb4980c04673a335d020a3ad3dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.294998 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a833fb4980c04673a335d020a3ad3dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.295684 8561 tablet_service.cc:1467] Processing CreateTablet for tablet f269f05bd0bf4bf5b520f90e646b0171 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.295926 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f269f05bd0bf4bf5b520f90e646b0171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.296902 11054 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.297075 8822 tablet_service.cc:1467] Processing CreateTablet for tablet f269f05bd0bf4bf5b520f90e646b0171 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.297274 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f269f05bd0bf4bf5b520f90e646b0171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.297637 8690 tablet_service.cc:1467] Processing CreateTablet for tablet a833fb4980c04673a335d020a3ad3dd3 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.297756 11056 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.297832 11054 tablet_bootstrap.cc:654] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.297866 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a833fb4980c04673a335d020a3ad3dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.298624 11056 tablet_bootstrap.cc:654] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.299640 6277 tablet_service.cc:1467] Processing CreateTablet for tablet e103e0e9cb16416dba7f8bbfc08ceac8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.299836 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e103e0e9cb16416dba7f8bbfc08ceac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.299966 8691 tablet_service.cc:1467] Processing CreateTablet for tablet f269f05bd0bf4bf5b520f90e646b0171 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.300176 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f269f05bd0bf4bf5b520f90e646b0171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.300624 11054 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:14.300693 11054 ts_tablet_manager.cc:1367] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:14.301134 11054 raft_consensus.cc:348] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.301220 11054 raft_consensus.cc:374] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.301245 11054 raft_consensus.cc:729] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.301365 11054 consensus_queue.cc:260] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.301628 11054 ts_tablet_manager.cc:1398] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.301769 11054 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.301878 11055 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:14.302083 11186 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:14.302740 11055 tablet_bootstrap.cc:654] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.302839 11056 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:14.302845 11054 tablet_bootstrap.cc:654] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.302911 11056 ts_tablet_manager.cc:1367] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:14.303325 11056 raft_consensus.cc:348] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.303422 11056 raft_consensus.cc:374] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.303452 11056 raft_consensus.cc:729] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.303560 11056 consensus_queue.cc:260] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.303529 6670 tablet_service.cc:1467] Processing CreateTablet for tablet e103e0e9cb16416dba7f8bbfc08ceac8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.303787 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e103e0e9cb16416dba7f8bbfc08ceac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.303798 11056 ts_tablet_manager.cc:1398] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.304167 11186 tablet_bootstrap.cc:654] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.304950 11054 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:14.305051 11054 ts_tablet_manager.cc:1367] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:14.305464 11054 raft_consensus.cc:348] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.305577 11054 raft_consensus.cc:374] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.305629 11054 raft_consensus.cc:729] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.305585 8957 tablet_service.cc:1467] Processing CreateTablet for tablet e103e0e9cb16416dba7f8bbfc08ceac8 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:14.305817 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e103e0e9cb16416dba7f8bbfc08ceac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.305853 11054 consensus_queue.cc:260] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.305972 11190 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:14.306730 11054 ts_tablet_manager.cc:1398] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.307205 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:14.309258 11112 raft_consensus.cc:1050] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:14.309438 11190 tablet_bootstrap.cc:654] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.309639 11112 consensus_queue.cc:237] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.310904 6691 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.311153 6428 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.311409 9277 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.311592 11186 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:14.311666 11186 ts_tablet_manager.cc:1367] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:14.311726 8842 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.311734 9277 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.312031 11186 raft_consensus.cc:348] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.312148 11186 raft_consensus.cc:374] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.312181 11186 raft_consensus.cc:729] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.312288 11186 consensus_queue.cc:260] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.312558 11186 ts_tablet_manager.cc:1398] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.313620 9963 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.313879 8575 raft_consensus.cc:1261] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.314327 9189 consensus_queue.cc:1035] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.314658 8843 raft_consensus.cc:1261] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.314968 9189 consensus_queue.cc:1035] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.316608 6157 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]). Expected version 1 got 0
I20240503 07:59:14.317085 10727 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.317341 10727 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:14.317763 6157 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]). Expected version 1 got 0
I20240503 07:59:14.319489 10729 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.319716 10729 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:14.321528 6150 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]). Expected version 1 got 0
I20240503 07:59:14.321801 6152 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0]). Expected version 1 got 0
I20240503 07:59:14.323765 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0] alter complete (version 1)
I20240503 07:59:14.326731 10810 ts_tablet_manager.cc:902] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:14.327194 10810 tablet_copy_client.cc:308] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:14.327579 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 5631f3e97fe04013a2a7140f44260003 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:56365)
I20240503 07:59:14.327663 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 5631f3e97fe04013a2a7140f44260003 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:56365: session id = c8d71787690245e681c85a087086e7ed-5631f3e97fe04013a2a7140f44260003
I20240503 07:59:14.328341 6450 tablet_copy_source_session.cc:205] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.329036 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5631f3e97fe04013a2a7140f44260003. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.329110 11199 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:14.329322 11055 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:14.329351 11199 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:14.329419 11055 ts_tablet_manager.cc:1367] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.000s
I20240503 07:59:14.329658 11183 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:14.329816 11055 raft_consensus.cc:348] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.329897 11055 raft_consensus.cc:374] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.329924 11055 raft_consensus.cc:729] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.330039 11055 consensus_queue.cc:260] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.330283 11055 ts_tablet_manager.cc:1398] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.330438 11055 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:14.330472 11183 tablet_bootstrap.cc:654] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.330891 10810 tablet_copy_client.cc:783] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.330996 10810 tablet_copy_client.cc:647] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.331213 11055 tablet_bootstrap.cc:654] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.332312 11190 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:14.332401 11190 ts_tablet_manager.cc:1367] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.000s
I20240503 07:59:14.332314 9277 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.333057 11190 raft_consensus.cc:348] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.333338 11190 raft_consensus.cc:374] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.333380 11190 raft_consensus.cc:729] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.333483 11190 consensus_queue.cc:260] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.333817 11190 ts_tablet_manager.cc:1398] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.334659 6428 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.334935 6157 catalog_manager.cc:5484] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.335819 8842 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.335836 6691 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.338694 11055 tablet_bootstrap.cc:492] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:14.338809 11055 ts_tablet_manager.cc:1367] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:14.339224 11055 raft_consensus.cc:348] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.339512 10810 tablet_copy_client.cc:523] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.339506 11055 raft_consensus.cc:374] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.339566 11055 raft_consensus.cc:729] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.339556 6560 consensus_queue.cc:237] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.339689 11055 consensus_queue.cc:260] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.340458 11055 ts_tablet_manager.cc:1398] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.340675 10810 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.340875 8842 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.341377 9963 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.341598 6691 raft_consensus.cc:1261] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.342207 9963 consensus_queue.cc:1035] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.343649 9278 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.344959 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 824c42be8152494fad2cbc76ece56981 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.345515 6691 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.345788 11183 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:14.345876 11183 ts_tablet_manager.cc:1367] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:14.346280 11183 raft_consensus.cc:348] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.346375 11183 raft_consensus.cc:374] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.346406 11183 raft_consensus.cc:729] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.346331 6152 catalog_manager.cc:5484] T 824c42be8152494fad2cbc76ece56981 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.346519 11183 consensus_queue.cc:260] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.346783 11183 ts_tablet_manager.cc:1398] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:14.347393 8842 raft_consensus.cc:2932] T 824c42be8152494fad2cbc76ece56981 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.347733 11183 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:14.348450 11183 tablet_bootstrap.cc:654] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.349931 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 824c42be8152494fad2cbc76ece56981 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.350100 9178 tablet_replica.cc:330] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.350211 9178 raft_consensus.cc:2229] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.350343 9178 raft_consensus.cc:2258] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.351306 9178 ts_tablet_manager.cc:1875] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.353260 11183 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:14.353251 9178 ts_tablet_manager.cc:1888] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.353309 9178 log.cc:1196] T 824c42be8152494fad2cbc76ece56981 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/824c42be8152494fad2cbc76ece56981
I20240503 07:59:14.353314 11183 ts_tablet_manager.cc:1367] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:14.353595 11183 raft_consensus.cc:348] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.353670 11183 raft_consensus.cc:374] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.353699 11183 raft_consensus.cc:729] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.353770 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 824c42be8152494fad2cbc76ece56981 (table rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]) successfully deleted
I20240503 07:59:14.353801 11183 consensus_queue.cc:260] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.354074 11183 ts_tablet_manager.cc:1398] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.354691 6691 raft_consensus.cc:1203] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:14.355280 11166 raft_consensus.cc:482] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.355358 11166 raft_consensus.cc:504] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.355660 11166 leader_election.cc:288] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:14.356079 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f269f05bd0bf4bf5b520f90e646b0171" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:14.356199 8575 raft_consensus.cc:2454] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:14.356427 8776 leader_election.cc:302] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:14.356614 11166 raft_consensus.cc:2781] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.356602 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f269f05bd0bf4bf5b520f90e646b0171" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:14.356652 11166 raft_consensus.cc:482] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.356683 11166 raft_consensus.cc:3037] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.356700 8712 raft_consensus.cc:2454] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:14.357524 11166 raft_consensus.cc:504] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.357805 11166 leader_election.cc:288] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:14.358074 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f269f05bd0bf4bf5b520f90e646b0171" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:14.358173 8712 raft_consensus.cc:3037] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.358711 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f269f05bd0bf4bf5b520f90e646b0171" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:14.358817 8575 raft_consensus.cc:3037] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.358947 8712 raft_consensus.cc:2454] T f269f05bd0bf4bf5b520f90e646b0171 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:14.359241 8779 leader_election.cc:302] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:14.359381 11166 raft_consensus.cc:2781] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.359445 11166 raft_consensus.cc:686] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:14.359549 11166 consensus_queue.cc:237] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:14.359644 8575 raft_consensus.cc:2454] T f269f05bd0bf4bf5b520f90e646b0171 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:14.361339 6156 catalog_manager.cc:5484] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.373606 10810 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.373867 10810 tablet_bootstrap.cc:492] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:14.374023 10810 ts_tablet_manager.cc:1367] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.000s
I20240503 07:59:14.374401 10810 raft_consensus.cc:348] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.374684 10810 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:14.374825 10810 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.376261 10810 ts_tablet_manager.cc:1398] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:14.376672 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-5631f3e97fe04013a2a7140f44260003 received from {username='slave'} at 127.0.99.133:56365
I20240503 07:59:14.376746 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session c8d71787690245e681c85a087086e7ed-5631f3e97fe04013a2a7140f44260003 on tablet 5631f3e97fe04013a2a7140f44260003 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:14.395466 8842 raft_consensus.cc:1203] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:14.413225 10401 ts_tablet_manager.cc:902] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:14.413694 10401 tablet_copy_client.cc:308] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:14.414214 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 39b01318dc1543cba2cc991e9db91c28 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:14.414320 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 39b01318dc1543cba2cc991e9db91c28 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-39b01318dc1543cba2cc991e9db91c28
I20240503 07:59:14.415009 6712 tablet_copy_source_session.cc:205] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.415763 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b01318dc1543cba2cc991e9db91c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.417418 10810 ts_tablet_manager.cc:902] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:14.417609 10401 tablet_copy_client.cc:783] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.417742 10401 tablet_copy_client.cc:647] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.418109 10810 tablet_copy_client.cc:308] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:14.418587 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 60f643dce3eb4c669b36b0714a31651b from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:56365)
I20240503 07:59:14.418680 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 60f643dce3eb4c669b36b0714a31651b from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:56365: session id = c8d71787690245e681c85a087086e7ed-60f643dce3eb4c669b36b0714a31651b
I20240503 07:59:14.419449 6450 tablet_copy_source_session.cc:205] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.419759 10401 tablet_copy_client.cc:523] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.421041 10401 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.423204 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f643dce3eb4c669b36b0714a31651b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.425197 10810 tablet_copy_client.cc:783] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.425328 10810 tablet_copy_client.cc:647] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.426903 10810 tablet_copy_client.cc:523] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.427223 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:14.427920 10810 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.432152 9400 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.432245 9400 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.432556 9400 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:14.433367 10727 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.433682 10727 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:14.433627 6560 raft_consensus.cc:1203] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:14.434927 11199 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.435168 11199 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:14.436419 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:14.436550 8977 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:14.436839 8647 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:14.437172 10729 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:14.437227 9189 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.437279 9189 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.437310 9189 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.437426 10729 tablet.cc:1620] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:14.438071 9189 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.438360 9189 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:14.440807 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:14.440935 8842 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:14.441681 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:14.441785 8842 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.441880 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:14.441983 8977 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.442597 8842 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:14.442799 8977 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:14.442898 8647 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:14.443143 9400 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.443214 9400 raft_consensus.cc:686] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:14.443329 9400 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:14.444484 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=06e7e89181f04ed38f5a6d02116047a0] alter complete (version 2)
I20240503 07:59:14.446255 6156 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.451333 6691 consensus_queue.cc:237] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:14.452584 8842 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:14.452988 6628 consensus_peers.cc:485] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4e4ca6638764fb49f3afcab7688e2b3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.452992 8977 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:14.453097 10254 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.453358 8077 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:14.454838 8077 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:14.455657 8977 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:14.457504 6156 catalog_manager.cc:5484] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.458598 8842 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:14.467545 10401 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.467778 10401 tablet_bootstrap.cc:492] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:14.467928 10401 ts_tablet_manager.cc:1367] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.001s
I20240503 07:59:14.468310 10401 raft_consensus.cc:348] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.468415 10401 raft_consensus.cc:729] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:14.468505 10401 consensus_queue.cc:260] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.469218 10401 ts_tablet_manager.cc:1398] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.469571 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-39b01318dc1543cba2cc991e9db91c28 received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:14.469631 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-39b01318dc1543cba2cc991e9db91c28 on tablet 39b01318dc1543cba2cc991e9db91c28 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:14.480254 10810 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.480487 10810 tablet_bootstrap.cc:492] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:14.480654 10810 ts_tablet_manager.cc:1367] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.007s
I20240503 07:59:14.481112 10810 raft_consensus.cc:348] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.481225 10810 raft_consensus.cc:729] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:14.481307 10810 consensus_queue.cc:260] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.481551 10810 ts_tablet_manager.cc:1398] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:14.481947 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-60f643dce3eb4c669b36b0714a31651b received from {username='slave'} at 127.0.99.133:56365
I20240503 07:59:14.482014 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session c8d71787690245e681c85a087086e7ed-60f643dce3eb4c669b36b0714a31651b on tablet 60f643dce3eb4c669b36b0714a31651b with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:14.548835 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:14.553272 10727 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.553813 11199 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.554016 10729 tablet.cc:1722] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:14.558312 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=06e7e89181f04ed38f5a6d02116047a0] alter complete (version 3)
I20240503 07:59:14.574182 9347 raft_consensus.cc:1050] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:14.574527 9347 consensus_queue.cc:237] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.575649 8977 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.575685 6296 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.575749 6560 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.576031 8296 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.576467 8296 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.576661 9968 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.578394 9347 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.578733 6296 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.578824 6560 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.580010 10239 raft_consensus.cc:1050] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:14.580267 8977 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.580367 10239 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:14.581002 9803 raft_consensus.cc:482] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:14.581084 9803 raft_consensus.cc:504] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.581352 9803 leader_election.cc:288] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:14.581621 8575 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:14.582126 6428 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:14.582369 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e103e0e9cb16416dba7f8bbfc08ceac8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:14.582471 8976 raft_consensus.cc:2454] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:14.582342 6157 catalog_manager.cc:5484] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.582679 11370 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:14.582726 6234 leader_election.cc:302] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:14.582962 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e103e0e9cb16416dba7f8bbfc08ceac8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:14.583076 6691 raft_consensus.cc:2454] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:14.583110 11371 raft_consensus.cc:2781] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:14.583196 11371 raft_consensus.cc:482] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:14.583261 11371 raft_consensus.cc:3037] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.583380 9803 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:14.584527 8977 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:14.584605 11371 raft_consensus.cc:504] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.584920 11371 leader_election.cc:288] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:14.585567 6691 consensus_queue.cc:237] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.586558 11371 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:14.586625 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e103e0e9cb16416dba7f8bbfc08ceac8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:14.586706 6690 raft_consensus.cc:3037] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.586725 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e103e0e9cb16416dba7f8bbfc08ceac8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:14.586835 8976 raft_consensus.cc:3037] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:14.587410 6690 raft_consensus.cc:2454] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:14.587589 8976 raft_consensus.cc:2454] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:14.587872 6234 leader_election.cc:302] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:14.588013 11371 raft_consensus.cc:2781] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:14.588091 11371 raft_consensus.cc:686] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:14.588258 11371 consensus_queue.cc:237] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:14.588750 6298 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.588934 8977 raft_consensus.cc:1261] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.589133 8077 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.589380 8296 consensus_queue.cc:1035] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.589932 6157 catalog_manager.cc:5484] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.590955 11371 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.591147 9968 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.591285 8977 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.591528 8976 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.592528 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2762b2e6de842bb8c898aea5b3910ba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.593051 8575 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.593523 6298 raft_consensus.cc:2932] T a2762b2e6de842bb8c898aea5b3910ba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.595877 6156 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.597311 6152 catalog_manager.cc:5484] T a2762b2e6de842bb8c898aea5b3910ba P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.598553 6428 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.601730 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet a2762b2e6de842bb8c898aea5b3910ba with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.601975 9176 tablet_replica.cc:330] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.602133 9176 raft_consensus.cc:2229] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.602286 9176 raft_consensus.cc:2258] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.603155 9176 ts_tablet_manager.cc:1875] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.605129 9176 ts_tablet_manager.cc:1888] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.605178 9176 log.cc:1196] T a2762b2e6de842bb8c898aea5b3910ba P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2762b2e6de842bb8c898aea5b3910ba
I20240503 07:59:14.605588 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet a2762b2e6de842bb8c898aea5b3910ba (table rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]) successfully deleted
I20240503 07:59:14.606987 6298 consensus_queue.cc:237] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:14.607995 8575 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:14.608037 8976 raft_consensus.cc:1261] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:14.608390 11371 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:14.608670 11370 consensus_queue.cc:1035] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:14.610780 11046 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.612177 8976 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.613155 6157 catalog_manager.cc:5484] T 2f943ecfe59f432c861a54b16c62572f P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.615099 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f943ecfe59f432c861a54b16c62572f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.615417 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f943ecfe59f432c861a54b16c62572f with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.615625 9178 tablet_replica.cc:330] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.615736 9178 raft_consensus.cc:2229] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.615898 9178 raft_consensus.cc:2258] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.615864 8575 raft_consensus.cc:2932] T 2f943ecfe59f432c861a54b16c62572f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:14.616868 9178 ts_tablet_manager.cc:1875] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.618647 9178 ts_tablet_manager.cc:1888] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:14.618695 9178 log.cc:1196] T 2f943ecfe59f432c861a54b16c62572f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f943ecfe59f432c861a54b16c62572f
I20240503 07:59:14.619115 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 2f943ecfe59f432c861a54b16c62572f (table rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]) successfully deleted
I20240503 07:59:14.634534 8842 raft_consensus.cc:1203] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:14.639317 11376 ts_tablet_manager.cc:902] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:14.639742 11376 tablet_copy_client.cc:308] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:14.640136 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet bfd4259777e5492c904e6d65a36d92ed from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:32897)
I20240503 07:59:14.640225 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet bfd4259777e5492c904e6d65a36d92ed from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:32897: session id = 9553fe3c4ccd4e91b7c176aad82fec09-bfd4259777e5492c904e6d65a36d92ed
I20240503 07:59:14.640969 6318 tablet_copy_source_session.cc:205] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.641688 11376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4259777e5492c904e6d65a36d92ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.643263 11376 tablet_copy_client.cc:783] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.643393 11376 tablet_copy_client.cc:647] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.644348 11376 tablet_copy_client.cc:523] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.645301 11376 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:14.660739 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.660880 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:14.662683 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 62555c2998e14f6e85896ab61f40e2dc
I20240503 07:59:14.663053 11376 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.663172 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 62555c2998e14f6e85896ab61f40e2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:14.663318 11376 tablet_bootstrap.cc:492] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:14.663461 11382 tablet_replica.cc:330] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:14.663472 11376 ts_tablet_manager.cc:1367] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20240503 07:59:14.663542 8561 tablet_service.cc:1514] Processing DeleteTablet for tablet 62555c2998e14f6e85896ab61f40e2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:14.663318 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet 62555c2998e14f6e85896ab61f40e2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:14.663635 11382 raft_consensus.cc:2229] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.663707 10259 tablet_replica.cc:330] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:14.663758 11382 raft_consensus.cc:2258] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.663810 10259 raft_consensus.cc:2229] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.663960 10259 raft_consensus.cc:2258] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.663928 11376 raft_consensus.cc:348] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:14.664022 11376 raft_consensus.cc:729] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:14.664109 11376 consensus_queue.cc:260] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:14.664253 11382 ts_tablet_manager.cc:1875] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.664216 11383 tablet_replica.cc:330] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:14.664397 11376 ts_tablet_manager.cc:1398] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.664405 11383 raft_consensus.cc:2229] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.664538 10259 ts_tablet_manager.cc:1875] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.664574 11383 raft_consensus.cc:2258] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.664837 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-bfd4259777e5492c904e6d65a36d92ed received from {username='slave'} at 127.0.99.134:32897
I20240503 07:59:14.664893 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-bfd4259777e5492c904e6d65a36d92ed on tablet bfd4259777e5492c904e6d65a36d92ed with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:14.665405 11383 ts_tablet_manager.cc:1875] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.666790 11382 ts_tablet_manager.cc:1888] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.666857 10254 raft_consensus.cc:1050] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:14.666927 11382 log.cc:1196] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/62555c2998e14f6e85896ab61f40e2dc
I20240503 07:59:14.667165 11382 ts_tablet_manager.cc:1909] T 62555c2998e14f6e85896ab61f40e2dc P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:14.667264 10254 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.667708 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table_new_name [id=06e7e89181f04ed38f5a6d02116047a0]) successfully deleted
I20240503 07:59:14.668054 10259 ts_tablet_manager.cc:1888] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.668112 10259 log.cc:1196] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/62555c2998e14f6e85896ab61f40e2dc
I20240503 07:59:14.668323 10259 ts_tablet_manager.cc:1909] T 62555c2998e14f6e85896ab61f40e2dc P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:14.668725 8976 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.668967 6428 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.669085 6560 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:14.669176 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table_new_name [id=06e7e89181f04ed38f5a6d02116047a0]) successfully deleted
I20240503 07:59:14.669363 9347 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.669513 8077 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.669827 9347 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:14.669940 11383 ts_tablet_manager.cc:1888] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.670002 11383 log.cc:1196] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/62555c2998e14f6e85896ab61f40e2dc
I20240503 07:59:14.670797 11383 ts_tablet_manager.cc:1909] T 62555c2998e14f6e85896ab61f40e2dc P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:14.671458 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 62555c2998e14f6e85896ab61f40e2dc (table rebalancer_dynamic_table_new_name [id=06e7e89181f04ed38f5a6d02116047a0]) successfully deleted
I20240503 07:59:14.671526 9347 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.671998 6428 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.672726 6560 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.673532 8976 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.674157 6157 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.678869 6691 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:14.680027 6428 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.680366 8296 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.680891 8976 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:14.681243 8296 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:14.682163 8296 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.682466 6428 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.683485 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8861b52c306e43d8b0615202cf7d8e35 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.684446 8976 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:14.686234 6152 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.688264 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 8861b52c306e43d8b0615202cf7d8e35 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.688499 9176 tablet_replica.cc:330] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.688637 9176 raft_consensus.cc:2229] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.688860 9176 raft_consensus.cc:2258] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.690155 9176 ts_tablet_manager.cc:1875] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.691902 9176 ts_tablet_manager.cc:1888] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:14.691969 9176 log.cc:1196] T 8861b52c306e43d8b0615202cf7d8e35 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8861b52c306e43d8b0615202cf7d8e35
I20240503 07:59:14.692461 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 8861b52c306e43d8b0615202cf7d8e35 (table rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]) successfully deleted
I20240503 07:59:14.696323 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 07:59:14.696450 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 07:59:14.698279 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 4 replicas of tablet b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.698724 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.698750 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.698913 9178 tablet_replica.cc:330] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.698910 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:14.699038 9178 raft_consensus.cc:2229] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.699072 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e0f5369e8349eba7188b39ae6bac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.699107 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:14.699169 9178 raft_consensus.cc:2258] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.699231 11382 tablet_replica.cc:330] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:14.699366 11382 raft_consensus.cc:2229] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Raft consensus shutting down.
I20240503 07:59:14.699465 11382 raft_consensus.cc:2258] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Raft consensus is shut down!
I20240503 07:59:14.699637 9178 ts_tablet_manager.cc:1875] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.699672 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d855e5edf05471983dfdbd40f37560f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.699762 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 77cc5703d29445f691675b12fc1ce191
I20240503 07:59:14.699975 11382 ts_tablet_manager.cc:1875] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.700009 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d855e5edf05471983dfdbd40f37560f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:14.700382 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cc5703d29445f691675b12fc1ce191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.700814 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cc5703d29445f691675b12fc1ce191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:14.700865 8561 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d855e5edf05471983dfdbd40f37560f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:14.701045 11383 tablet_replica.cc:330] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:14.701140 11383 raft_consensus.cc:2229] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.701243 11383 raft_consensus.cc:2258] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.701678 11383 ts_tablet_manager.cc:1875] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.701692 9178 ts_tablet_manager.cc:1888] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.701750 9178 log.cc:1196] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.702003 9178 ts_tablet_manager.cc:1909] T b2e0f5369e8349eba7188b39ae6bac30 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:14.702090 11382 ts_tablet_manager.cc:1888] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.702144 11382 log.cc:1196] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.702373 11382 ts_tablet_manager.cc:1909] T b2e0f5369e8349eba7188b39ae6bac30 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:14.702533 9178 tablet_replica.cc:330] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:14.702534 9176 tablet_replica.cc:330] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:14.702636 9176 raft_consensus.cc:2229] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.702636 9178 raft_consensus.cc:2229] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.702668 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet b2e0f5369e8349eba7188b39ae6bac30 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.702800 9178 raft_consensus.cc:2258] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.702808 9176 raft_consensus.cc:2258] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.703220 9176 ts_tablet_manager.cc:1875] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.703217 9178 ts_tablet_manager.cc:1875] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.703290 11383 ts_tablet_manager.cc:1888] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.703336 11383 log.cc:1196] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:14.703498 11383 ts_tablet_manager.cc:1909] T 7d855e5edf05471983dfdbd40f37560f P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:14.704063 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 7d855e5edf05471983dfdbd40f37560f (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.704872 9175 tablet_replica.cc:330] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.704999 9175 raft_consensus.cc:2229] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:14.705024 9176 ts_tablet_manager.cc:1888] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.705094 9176 log.cc:1196] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:14.705258 9175 raft_consensus.cc:2258] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.705260 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet b2e0f5369e8349eba7188b39ae6bac30 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.705313 9176 ts_tablet_manager.cc:1909] T 7d855e5edf05471983dfdbd40f37560f P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:14.705731 9175 ts_tablet_manager.cc:1875] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.705862 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 7d855e5edf05471983dfdbd40f37560f (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.707139 9175 ts_tablet_manager.cc:1888] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.707182 9175 log.cc:1196] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.707332 9175 ts_tablet_manager.cc:1909] T b2e0f5369e8349eba7188b39ae6bac30 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:14.707470 9177 tablet_replica.cc:330] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.707571 9177 raft_consensus.cc:2229] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.707685 9177 raft_consensus.cc:2258] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.708040 9177 ts_tablet_manager.cc:1875] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.708993 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet b2e0f5369e8349eba7188b39ae6bac30 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.709123 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 77cc5703d29445f691675b12fc1ce191 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:14 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:14.709280 9175 tablet_replica.cc:330] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:14.709393 9175 raft_consensus.cc:2229] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.709491 9178 ts_tablet_manager.cc:1888] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.709510 9175 raft_consensus.cc:2258] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.709559 9178 log.cc:1196] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/77cc5703d29445f691675b12fc1ce191
I20240503 07:59:14.709808 9177 ts_tablet_manager.cc:1888] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:14.709884 9177 log.cc:1196] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b2e0f5369e8349eba7188b39ae6bac30
I20240503 07:59:14.709903 9175 ts_tablet_manager.cc:1875] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.710070 9178 ts_tablet_manager.cc:1909] T 77cc5703d29445f691675b12fc1ce191 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:14.710141 9177 ts_tablet_manager.cc:1909] T b2e0f5369e8349eba7188b39ae6bac30 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:14.710628 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 77cc5703d29445f691675b12fc1ce191 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.710647 9177 tablet_replica.cc:330] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.710745 9177 raft_consensus.cc:2229] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.710904 9177 raft_consensus.cc:2258] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.711050 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet b2e0f5369e8349eba7188b39ae6bac30 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.711308 9177 ts_tablet_manager.cc:1875] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.711592 9175 ts_tablet_manager.cc:1888] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.711647 9175 log.cc:1196] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/77cc5703d29445f691675b12fc1ce191
I20240503 07:59:14.711818 9175 ts_tablet_manager.cc:1909] T 77cc5703d29445f691675b12fc1ce191 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:14.712625 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 77cc5703d29445f691675b12fc1ce191 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.712723 9177 ts_tablet_manager.cc:1888] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.6
I20240503 07:59:14.712800 9177 log.cc:1196] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d855e5edf05471983dfdbd40f37560f
I20240503 07:59:14.713166 9177 ts_tablet_manager.cc:1909] T 7d855e5edf05471983dfdbd40f37560f P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:14.713563 9177 tablet_replica.cc:330] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.713624 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 7d855e5edf05471983dfdbd40f37560f (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.713657 9177 raft_consensus.cc:2229] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.713745 9177 raft_consensus.cc:2258] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.714100 9177 ts_tablet_manager.cc:1875] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:14.715301 9177 ts_tablet_manager.cc:1888] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 07:59:14.715359 9177 log.cc:1196] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/77cc5703d29445f691675b12fc1ce191
I20240503 07:59:14.715529 9177 ts_tablet_manager.cc:1909] T 77cc5703d29445f691675b12fc1ce191 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:14.715989 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 77cc5703d29445f691675b12fc1ce191 (table rebalance_test_table_18 [id=1558bad4bc7b4dd99890cbbd33e36d27]) successfully deleted
I20240503 07:59:14.727793 10810 ts_tablet_manager.cc:902] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:14.728184 10810 tablet_copy_client.cc:308] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:14.728623 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet f635e30fb6974fa1a79d8a46eecc3d5e from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:44995)
I20240503 07:59:14.728713 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet f635e30fb6974fa1a79d8a46eecc3d5e from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:44995: session id = c8d71787690245e681c85a087086e7ed-f635e30fb6974fa1a79d8a46eecc3d5e
I20240503 07:59:14.729475 8732 tablet_copy_source_session.cc:205] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.730278 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f635e30fb6974fa1a79d8a46eecc3d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.731740 10810 tablet_copy_client.cc:783] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.731871 10810 tablet_copy_client.cc:647] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.732792 10810 tablet_copy_client.cc:523] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.733631 10810 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.736553 10810 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.736735 10810 tablet_bootstrap.cc:492] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:14.736896 10810 ts_tablet_manager.cc:1367] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s
I20240503 07:59:14.737236 10810 raft_consensus.cc:348] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.737326 10810 raft_consensus.cc:729] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:14.737408 10810 consensus_queue.cc:260] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:14.737620 10810 ts_tablet_manager.cc:1398] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:14.738015 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-f635e30fb6974fa1a79d8a46eecc3d5e received from {username='slave'} at 127.0.99.133:44995
I20240503 07:59:14.738078 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session c8d71787690245e681c85a087086e7ed-f635e30fb6974fa1a79d8a46eecc3d5e on tablet f635e30fb6974fa1a79d8a46eecc3d5e with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:14.764024 10401 ts_tablet_manager.cc:902] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:14.764242 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:14.764691 10401 tablet_copy_client.cc:308] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:14.765172 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 2d3f03610866454595f4b6b87b7cd9cb from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:50641)
I20240503 07:59:14.765260 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 2d3f03610866454595f4b6b87b7cd9cb from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:50641: session id = 616fdda6ba714cb7be01a564de2b9db2-2d3f03610866454595f4b6b87b7cd9cb
I20240503 07:59:14.766055 6712 tablet_copy_source_session.cc:205] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.766884 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3f03610866454595f4b6b87b7cd9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.768882 10401 tablet_copy_client.cc:783] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.769026 10401 tablet_copy_client.cc:647] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.769308 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]), partition=
I20240503 07:59:14.769544 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b087008d24472ba3e5e644c2bba6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.770327 10401 tablet_copy_client.cc:523] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.771426 10401 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:14.771539 11055 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:14.772367 11055 tablet_bootstrap.cc:654] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.772358 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]), partition=
I20240503 07:59:14.772575 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b087008d24472ba3e5e644c2bba6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.773046 8561 tablet_service.cc:1467] Processing CreateTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]), partition=
I20240503 07:59:14.773258 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b087008d24472ba3e5e644c2bba6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.774271 11418 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:14.775110 11056 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:14.775954 11056 tablet_bootstrap.cc:654] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.776582 11055 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:14.776676 11055 ts_tablet_manager.cc:1367] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:14.777084 11418 tablet_bootstrap.cc:654] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:14.777110 11055 raft_consensus.cc:348] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.777216 11055 raft_consensus.cc:374] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.777244 11055 raft_consensus.cc:729] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.777369 11055 consensus_queue.cc:260] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.777626 11055 ts_tablet_manager.cc:1398] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:14.778579 11418 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:14.778661 11418 ts_tablet_manager.cc:1367] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:14.779044 11418 raft_consensus.cc:348] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.779138 11418 raft_consensus.cc:374] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.779167 11418 raft_consensus.cc:729] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.779280 11418 consensus_queue.cc:260] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.779583 11418 ts_tablet_manager.cc:1398] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.780579 11056 tablet_bootstrap.cc:492] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:14.780671 11056 ts_tablet_manager.cc:1367] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:14.781126 11056 raft_consensus.cc:348] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.781225 11056 raft_consensus.cc:374] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:14.781255 11056 raft_consensus.cc:729] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:14.781363 11056 consensus_queue.cc:260] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:14.781630 11056 ts_tablet_manager.cc:1398] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:14.797567 10401 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.797832 10401 tablet_bootstrap.cc:492] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:14.798056 10401 ts_tablet_manager.cc:1367] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20240503 07:59:14.798454 10401 raft_consensus.cc:348] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.798556 10401 raft_consensus.cc:729] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:14.798643 10401 consensus_queue.cc:260] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:14.798903 10401 ts_tablet_manager.cc:1398] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:14.799305 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-2d3f03610866454595f4b6b87b7cd9cb received from {username='slave'} at 127.0.99.135:50641
I20240503 07:59:14.799392 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-2d3f03610866454595f4b6b87b7cd9cb on tablet 2d3f03610866454595f4b6b87b7cd9cb with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:14.808722 11448 ts_tablet_manager.cc:902] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:14.809553 11448 tablet_copy_client.cc:308] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:39417
I20240503 07:59:14.810133 8864 tablet_copy_service.cc:140] P 616fdda6ba714cb7be01a564de2b9db2: Received BeginTabletCopySession request for tablet 5060df68b95e48a8a849e77ba926e608 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:57073)
I20240503 07:59:14.810245 8864 tablet_copy_service.cc:161] P 616fdda6ba714cb7be01a564de2b9db2: Beginning new tablet copy session on tablet 5060df68b95e48a8a849e77ba926e608 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:57073: session id = ad910f92ef9d4db8a8962cecba289f8a-5060df68b95e48a8a849e77ba926e608
I20240503 07:59:14.810288 8575 raft_consensus.cc:1203] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:14.811555 8864 tablet_copy_source_session.cc:205] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.812593 11448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5060df68b95e48a8a849e77ba926e608. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.814849 11448 tablet_copy_client.cc:783] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.815006 11448 tablet_copy_client.cc:647] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.816183 11448 tablet_copy_client.cc:523] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.818070 11448 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:14.821311 11448 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.821597 11448 tablet_bootstrap.cc:492] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:14.821748 11448 ts_tablet_manager.cc:1367] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 07:59:14.822120 11448 raft_consensus.cc:348] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:14.822229 11448 raft_consensus.cc:729] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:14.822356 11448 consensus_queue.cc:260] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:14.822782 11448 ts_tablet_manager.cc:1398] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:14.823309 8864 tablet_copy_service.cc:342] P 616fdda6ba714cb7be01a564de2b9db2: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-5060df68b95e48a8a849e77ba926e608 received from {username='slave'} at 127.0.99.131:57073
I20240503 07:59:14.823380 8864 tablet_copy_service.cc:433] P 616fdda6ba714cb7be01a564de2b9db2: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-5060df68b95e48a8a849e77ba926e608 on tablet 5060df68b95e48a8a849e77ba926e608 with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:14.825526 10815 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.831710 9400 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.846581 9400 raft_consensus.cc:1050] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:14.846968 9400 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:14.848140 8575 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.848274 6560 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.848325 8976 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.848582 10079 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.848721 9189 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.849025 9189 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.850615 10815 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.851107 8976 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.851125 8575 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.853520 6156 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.853634 6560 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.857406 8712 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:14.857765 6691 consensus_queue.cc:237] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:14.859369 6428 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:14.859516 6627 consensus_peers.cc:485] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d -> Peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Couldn't send request to peer 3d08a06d53b14381bee6223c4e1a6027. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 558c0eb4a4334dbe91ac60fad4e820dc. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:14.859531 8575 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.859836 8296 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.859938 6560 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.860625 6298 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.860898 10079 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.861192 10079 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.861312 8296 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.861876 11240 raft_consensus.cc:1050] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:14.862311 11240 consensus_queue.cc:237] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:14.862900 9400 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.863641 6691 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.863730 6296 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.863811 8579 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.864176 11240 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.864282 11495 ts_tablet_manager.cc:902] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:14.864362 6560 raft_consensus.cc:2932] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:14.864514 11208 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.865615 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f534eefb7c334889a9c5d02175ae2d94 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.866286 6152 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:14.866823 8296 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:14.867211 8575 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.867153 6298 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:14.868603 9196 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.870316 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet f534eefb7c334889a9c5d02175ae2d94 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:14.870608 11501 tablet_replica.cc:330] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:14.870720 11501 raft_consensus.cc:2229] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.870839 11501 raft_consensus.cc:2258] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.870911 6157 catalog_manager.cc:5484] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.871308 11501 ts_tablet_manager.cc:1875] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.871376 6428 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } } }
I20240503 07:59:14.873503 11501 ts_tablet_manager.cc:1888] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:14.873615 11501 log.cc:1196] T f534eefb7c334889a9c5d02175ae2d94 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f534eefb7c334889a9c5d02175ae2d94
I20240503 07:59:14.874336 10916 consensus_queue.cc:1035] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.874670 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet f534eefb7c334889a9c5d02175ae2d94 (table rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]) successfully deleted
I20240503 07:59:14.878623 11495 tablet_copy_client.cc:308] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:14.878794 10916 consensus_queue.cc:1035] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:14.879274 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet f4e4ca6638764fb49f3afcab7688e2b3 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:45383)
I20240503 07:59:14.879366 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet f4e4ca6638764fb49f3afcab7688e2b3 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:45383: session id = d1f36d57661b4429a4be8d10b58d8ff7-f4e4ca6638764fb49f3afcab7688e2b3
I20240503 07:59:14.880213 6712 tablet_copy_source_session.cc:205] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:14.881464 9196 raft_consensus.cc:1050] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:14.881817 11495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e4ca6638764fb49f3afcab7688e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:14.881803 9196 consensus_queue.cc:237] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.883036 8712 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.883211 8579 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.883298 8842 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:14.883605 11208 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.883881 11208 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.884748 11495 tablet_copy_client.cc:783] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:14.884963 11495 tablet_copy_client.cc:647] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:14.885911 11240 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:14.886230 11240 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.886632 11495 tablet_copy_client.cc:523] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:14.887976 11495 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:14.887979 11240 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.890007 6296 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.890007 8842 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.890172 6152 catalog_manager.cc:5484] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.890988 8579 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.891685 8575 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.892347 6157 catalog_manager.cc:5484] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.895337 11495 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:14.895610 11495 tablet_bootstrap.cc:492] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:14.896003 8575 raft_consensus.cc:1203] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:14.896641 11495 ts_tablet_manager.cc:1367] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 07:59:14.897280 11495 raft_consensus.cc:348] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:14.898200 11495 raft_consensus.cc:729] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:14.898478 11495 consensus_queue.cc:260] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:14.898607 8712 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.899055 6691 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.899395 11495 ts_tablet_manager.cc:1398] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:14.900787 8976 consensus_queue.cc:237] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:14.901113 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-f4e4ca6638764fb49f3afcab7688e2b3 received from {username='slave'} at 127.0.99.130:45383
I20240503 07:59:14.901183 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-f4e4ca6638764fb49f3afcab7688e2b3 on tablet f4e4ca6638764fb49f3afcab7688e2b3 with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:14.901749 8842 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.902868 11522 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.904034 8575 raft_consensus.cc:1261] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.904357 11497 consensus_queue.cc:1035] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.906352 8842 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.907493 8575 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.908196 8976 consensus_queue.cc:237] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:14.908457 11522 raft_consensus.cc:2932] T a01fa99be75f498a8394aabbfabb3f5a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.909253 6150 catalog_manager.cc:5484] T a01fa99be75f498a8394aabbfabb3f5a P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:14.910614 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a01fa99be75f498a8394aabbfabb3f5a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.910753 6691 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.910768 8575 raft_consensus.cc:1261] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:14.911056 11240 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.911218 11208 consensus_queue.cc:1035] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:14.912348 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet a01fa99be75f498a8394aabbfabb3f5a with delete_type TABLET_DATA_TOMBSTONED (TS 9553fe3c4ccd4e91b7c176aad82fec09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:14.912606 10259 tablet_replica.cc:330] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:14.912758 10259 raft_consensus.cc:2229] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.912894 10259 raft_consensus.cc:2258] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.913451 11208 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.913604 10259 ts_tablet_manager.cc:1875] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.914525 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 94d5216ff8654561b44a0cfd0f9b1625 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:14.915475 10259 ts_tablet_manager.cc:1888] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:14.915532 10259 log.cc:1196] T a01fa99be75f498a8394aabbfabb3f5a P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a01fa99be75f498a8394aabbfabb3f5a
I20240503 07:59:14.915781 6691 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.915982 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet a01fa99be75f498a8394aabbfabb3f5a (table rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]) successfully deleted
I20240503 07:59:14.916230 8575 raft_consensus.cc:2932] T 94d5216ff8654561b44a0cfd0f9b1625 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:14.916700 6150 catalog_manager.cc:5484] T 94d5216ff8654561b44a0cfd0f9b1625 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 3 to 4, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:14.918990 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d5216ff8654561b44a0cfd0f9b1625 with delete_type TABLET_DATA_TOMBSTONED (TS c167a0a433d34b74affed13cbf96e302 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:14.919189 9177 tablet_replica.cc:330] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:14.919317 9177 raft_consensus.cc:2229] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:14.919508 9177 raft_consensus.cc:2258] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:14.920065 9177 ts_tablet_manager.cc:1875] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:14.922081 9177 ts_tablet_manager.cc:1888] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:14.922156 9177 log.cc:1196] T 94d5216ff8654561b44a0cfd0f9b1625 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/94d5216ff8654561b44a0cfd0f9b1625
I20240503 07:59:14.922603 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 94d5216ff8654561b44a0cfd0f9b1625 (table rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]) successfully deleted
I20240503 07:59:14.947659 8842 raft_consensus.cc:1203] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:15.018921 10815 raft_consensus.cc:482] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.019018 10815 raft_consensus.cc:504] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.019405 10815 leader_election.cc:288] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:15.019709 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b087008d24472ba3e5e644c2bba6a8" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:15.019737 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b087008d24472ba3e5e644c2bba6a8" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:15.019845 8575 raft_consensus.cc:2454] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:15.019873 6428 raft_consensus.cc:2454] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:15.020280 8645 leader_election.cc:302] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.020458 8712 raft_consensus.cc:1203] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:15.020521 10815 raft_consensus.cc:2781] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.020577 10815 raft_consensus.cc:482] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.020612 10815 raft_consensus.cc:3037] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.021476 10815 raft_consensus.cc:504] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.021788 10815 leader_election.cc:288] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:15.022466 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b087008d24472ba3e5e644c2bba6a8" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:15.022496 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b087008d24472ba3e5e644c2bba6a8" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:15.022578 8575 raft_consensus.cc:3037] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.022594 6428 raft_consensus.cc:3037] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.023522 8575 raft_consensus.cc:2454] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:15.023542 6428 raft_consensus.cc:2454] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:15.023758 8645 leader_election.cc:302] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.023939 10815 raft_consensus.cc:2781] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.024024 10815 raft_consensus.cc:686] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:15.024164 10815 consensus_queue.cc:237] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.025668 6157 catalog_manager.cc:5484] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.032438 10239 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.044147 9803 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.045722 8822 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:15.046360 8957 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 07:59:15.047650 8691 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:15.051043 6277 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:15.057878 6670 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:15.073235 8561 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 07:59:15.085039 6410 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:15.085054 6539 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:15.110255 6149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:15.116076 8957 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2ea716ca64e309efccd291db503d6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.116389 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2ea716ca64e309efccd291db503d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.117015 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 00bcbd5440ff4d56b1975edc4041b428 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.117264 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bcbd5440ff4d56b1975edc4041b428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.117303 8822 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2ea716ca64e309efccd291db503d6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.117535 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2ea716ca64e309efccd291db503d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.117581 8824 tablet_service.cc:1467] Processing CreateTablet for tablet edaf4aa5285f49c297ab412ceb179be9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.117808 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf4aa5285f49c297ab412ceb179be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.119474 11055 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:15.120157 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 00bcbd5440ff4d56b1975edc4041b428 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.120132 8955 tablet_service.cc:1467] Processing CreateTablet for tablet edaf4aa5285f49c297ab412ceb179be9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.120154 8561 tablet_service.cc:1467] Processing CreateTablet for tablet e1b2ea716ca64e309efccd291db503d6 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.120404 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bcbd5440ff4d56b1975edc4041b428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.120412 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2ea716ca64e309efccd291db503d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.121088 11570 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.121963 11570 tablet_bootstrap.cc:654] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.122367 11055 tablet_bootstrap.cc:654] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.122700 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 00bcbd5440ff4d56b1975edc4041b428 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.122882 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bcbd5440ff4d56b1975edc4041b428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.123399 11055 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:15.123441 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf4aa5285f49c297ab412ceb179be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.123481 11055 ts_tablet_manager.cc:1367] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:15.123940 11055 raft_consensus.cc:348] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.124038 11055 raft_consensus.cc:374] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.124076 11055 raft_consensus.cc:729] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.124178 11055 consensus_queue.cc:260] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.124436 11055 ts_tablet_manager.cc:1398] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.124806 6539 tablet_service.cc:1467] Processing CreateTablet for tablet edaf4aa5285f49c297ab412ceb179be9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.125041 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf4aa5285f49c297ab412ceb179be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.126936 11571 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:15.127028 11056 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.127269 11574 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:15.127800 11056 tablet_bootstrap.cc:654] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.128276 11574 tablet_bootstrap.cc:654] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.128424 11571 tablet_bootstrap.cc:654] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.128674 11056 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.128791 11056 ts_tablet_manager.cc:1367] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:15.129243 11056 raft_consensus.cc:348] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.129346 11056 raft_consensus.cc:374] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.129382 11056 raft_consensus.cc:729] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.129508 11571 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:15.129495 11056 consensus_queue.cc:260] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.129604 11571 ts_tablet_manager.cc:1367] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:15.129742 11056 ts_tablet_manager.cc:1398] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:15.129947 11056 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.130064 11571 raft_consensus.cc:348] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.130165 11571 raft_consensus.cc:374] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.130199 11571 raft_consensus.cc:729] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.130316 11571 consensus_queue.cc:260] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.130597 11571 ts_tablet_manager.cc:1398] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:15.130681 11056 tablet_bootstrap.cc:654] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.130749 11571 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:15.131522 11056 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.131599 11056 ts_tablet_manager.cc:1367] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.132071 11574 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:15.132222 11574 ts_tablet_manager.cc:1367] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:15.132287 11571 tablet_bootstrap.cc:654] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.133595 11574 raft_consensus.cc:348] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.133910 11056 raft_consensus.cc:348] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.134070 11056 raft_consensus.cc:374] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.134141 11056 raft_consensus.cc:729] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.134310 11056 consensus_queue.cc:260] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.134630 11056 ts_tablet_manager.cc:1398] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:15.133760 11574 raft_consensus.cc:374] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.136173 11574 raft_consensus.cc:729] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.137290 11574 consensus_queue.cc:260] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.137534 11570 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:15.137630 11570 ts_tablet_manager.cc:1367] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 07:59:15.137732 11574 ts_tablet_manager.cc:1398] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:15.138098 11570 raft_consensus.cc:348] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.138197 11570 raft_consensus.cc:374] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.138227 11570 raft_consensus.cc:729] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.138327 11570 consensus_queue.cc:260] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.138526 11571 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:15.138578 11570 ts_tablet_manager.cc:1398] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.138649 11571 ts_tablet_manager.cc:1367] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 07:59:15.138782 11570 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.139092 11571 raft_consensus.cc:348] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.139187 11571 raft_consensus.cc:374] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.139597 11570 tablet_bootstrap.cc:654] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.139221 11571 raft_consensus.cc:729] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.140136 11571 consensus_queue.cc:260] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.140547 11571 ts_tablet_manager.cc:1398] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:15.140872 11571 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:15.142041 11571 tablet_bootstrap.cc:654] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.143648 11571 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:15.143790 11571 ts_tablet_manager.cc:1367] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:15.144261 11571 raft_consensus.cc:348] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.144407 11571 raft_consensus.cc:374] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.144474 11571 raft_consensus.cc:729] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.144619 11571 consensus_queue.cc:260] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.144927 11571 ts_tablet_manager.cc:1398] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.149264 11570 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:15.149369 11570 ts_tablet_manager.cc:1367] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240503 07:59:15.149859 11570 raft_consensus.cc:348] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.149969 11570 raft_consensus.cc:374] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.150005 11570 raft_consensus.cc:729] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.150112 11570 consensus_queue.cc:260] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.150328 11570 ts_tablet_manager.cc:1398] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.159787 8575 consensus_queue.cc:237] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
W20240503 07:59:15.161465 8514 consensus_peers.cc:485] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5b4c776eac24f739d85e09707cd751b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.161731 8712 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.162113 6296 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.162142 10412 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.162588 10412 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.164439 10412 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.164731 8712 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.167227 6149 catalog_manager.cc:5484] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.168077 6296 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.172156 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:15.175838 6428 raft_consensus.cc:1261] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.175961 8575 raft_consensus.cc:1261] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.176383 10815 consensus_queue.cc:1035] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.177798 10815 consensus_queue.cc:1035] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.179111 6149 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]). Expected version 1 got 0
I20240503 07:59:15.180224 6149 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]). Expected version 1 got 0
I20240503 07:59:15.181128 11601 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.182343 6149 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]). Expected version 1 got 0
I20240503 07:59:15.182904 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]). Expected version 1 got 0
I20240503 07:59:15.183487 11601 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:15.183598 6149 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd]). Expected version 1 got 0
I20240503 07:59:15.185156 8712 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:15.185675 11603 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.185950 11603 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:15.186039 6157 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd] alter complete (version 1)
I20240503 07:59:15.186254 11602 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.186638 11602 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:15.186699 8842 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.187170 8575 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:15.187608 8645 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8cf3d6a3ace46769854ed61f340fb25. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.187763 10815 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.188009 9400 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.189697 9400 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:15.191278 8575 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:15.192222 6149 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.192463 8842 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:15.200902 6298 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.201493 8575 raft_consensus.cc:1203] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:15.202222 8575 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.202471 8712 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:15.202510 6235 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551f25dcee034b7894498e3c748b0f0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.202628 11371 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.202780 11370 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.204403 9803 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.204843 8712 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.204911 8575 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.207151 6155 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.214607 6428 consensus_queue.cc:237] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:15.215768 6560 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:15.215778 6691 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:15.215904 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 974c815ff8e94190a7bf28444cb3cd9f. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.216135 11449 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.216305 9288 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.217788 11427 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.218329 6560 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.218305 6691 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.220335 6149 catalog_manager.cc:5484] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.228093 6298 consensus_queue.cc:237] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.229171 6691 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.229282 8976 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:15.229403 6235 consensus_peers.cc:485] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e103e0e9cb16416dba7f8bbfc08ceac8. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.229492 11371 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.229882 11371 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.230701 11046 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.231130 6691 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.233281 6155 catalog_manager.cc:5484] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.237051 8976 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:15.242766 9288 raft_consensus.cc:1050] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:15.243122 9288 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.244501 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.244510 8976 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.244536 8575 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.244897 11427 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.245031 9288 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.245169 11449 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.247706 11427 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.248270 8977 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.249934 10079 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.250015 10079 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.250325 10079 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:15.250443 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.250600 6157 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.250906 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:15.251003 8575 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:15.251120 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:15.251226 8977 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:15.251294 8645 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.251832 10079 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.251883 10079 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.251927 10079 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.252727 10079 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.252840 8575 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.253077 10079 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:15.253139 6691 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:15.253564 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:15.253654 8579 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.254438 8579 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:15.255177 6560 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.256067 8645 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
W20240503 07:59:15.256183 6625 consensus_peers.cc:485] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82de209205b342cfaac10506fdac9303. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.256353 10254 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.256386 8842 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.256397 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:15.256490 8976 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.257305 8976 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:15.257833 10254 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.258176 8296 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.258958 10079 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.259045 10079 raft_consensus.cc:686] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:15.259161 10079 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:15.259739 6560 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.260695 6156 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.260846 6149 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.265640 9963 raft_consensus.cc:482] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.265722 9963 raft_consensus.cc:504] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.266017 9963 leader_election.cc:288] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:15.266376 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf4aa5285f49c297ab412ceb179be9" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:15.266486 8977 raft_consensus.cc:2454] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:15.266465 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf4aa5285f49c297ab412ceb179be9" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:15.266570 8843 raft_consensus.cc:2454] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:15.266754 6496 leader_election.cc:302] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:15.267226 9963 raft_consensus.cc:2781] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.267280 9963 raft_consensus.cc:482] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.267313 9963 raft_consensus.cc:3037] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.268184 9963 raft_consensus.cc:504] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.268482 9963 leader_election.cc:288] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:15.268890 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf4aa5285f49c297ab412ceb179be9" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:15.268905 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf4aa5285f49c297ab412ceb179be9" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:15.268981 8977 raft_consensus.cc:3037] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.269026 8843 raft_consensus.cc:3037] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.269765 8977 raft_consensus.cc:2454] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:15.269843 8843 raft_consensus.cc:2454] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:15.270052 6496 leader_election.cc:302] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:15.270471 9963 raft_consensus.cc:2781] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.270612 9963 raft_consensus.cc:686] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:15.270737 9963 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.272415 8842 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.272516 6155 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.288455 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:15.291544 11521 raft_consensus.cc:482] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.291632 11521 raft_consensus.cc:504] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.292562 11521 leader_election.cc:288] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:15.293143 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2ea716ca64e309efccd291db503d6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:15.293275 8579 raft_consensus.cc:2454] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:15.293787 8910 leader_election.cc:302] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.294106 11521 raft_consensus.cc:2781] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.294162 11521 raft_consensus.cc:482] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.294196 11521 raft_consensus.cc:3037] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.294142 11601 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.294401 11601 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:15.294651 11602 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.294859 11602 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:15.295087 11521 raft_consensus.cc:504] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.295306 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2ea716ca64e309efccd291db503d6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:15.295377 11521 leader_election.cc:288] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:15.295410 8842 raft_consensus.cc:2454] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:15.295907 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2ea716ca64e309efccd291db503d6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:15.296274 8579 raft_consensus.cc:3037] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.297168 8579 raft_consensus.cc:2454] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:15.297470 8910 leader_election.cc:302] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.297461 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b2ea716ca64e309efccd291db503d6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:15.297567 8842 raft_consensus.cc:3037] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.297756 11521 raft_consensus.cc:2781] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.297835 11521 raft_consensus.cc:686] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:15.297947 11521 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.298436 8842 raft_consensus.cc:2454] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:15.299624 6149 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=af669f53d59f4eb4bdc4b498325400dd] alter complete (version 2)
I20240503 07:59:15.301293 6149 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.301617 11603 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.301892 11603 tablet.cc:1620] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:15.338867 6428 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47576
I20240503 07:59:15.338984 6428 raft_consensus.cc:595] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:15.346755 8842 raft_consensus.cc:1203] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:15.352157 6428 raft_consensus.cc:1203] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:15.371647 6560 raft_consensus.cc:1203] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:15.380892 8712 consensus_queue.cc:237] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:15.382167 6691 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.382185 8842 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:15.382294 8645 consensus_peers.cc:485] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09866b93f75545a19c9451556254da48. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.382535 9400 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.382701 10079 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:15.384130 10815 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.384541 8842 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.384613 6691 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:15.386893 6149 catalog_manager.cc:5484] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.393719 6560 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:15.394934 6691 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.395047 8579 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:15.395078 6496 consensus_peers.cc:485] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c597bc1a19274f55b9d60558db35fcae. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.395264 11755 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.395427 11648 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.396943 11755 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:15.397323 6691 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:15.397459 8579 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:15.399811 6155 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:15.403065 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:15.407518 11601 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.407864 11603 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.410671 11602 tablet.cc:1722] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:15.412107 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=af669f53d59f4eb4bdc4b498325400dd] alter complete (version 3)
I20240503 07:59:15.446657 9288 raft_consensus.cc:1050] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:15.447021 9288 consensus_queue.cc:237] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.448161 6691 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.448518 11427 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.448704 8579 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.448993 6560 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.450208 11427 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.450393 11449 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.453177 11427 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.453584 6560 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.455471 6156 catalog_manager.cc:5484] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.455550 6691 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.460141 8579 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.461442 11812 ts_tablet_manager.cc:902] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:15.461901 11812 tablet_copy_client.cc:308] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:15.462045 6428 consensus_queue.cc:237] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.462333 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 558c0eb4a4334dbe91ac60fad4e820dc from peer 3d08a06d53b14381bee6223c4e1a6027 ({username='slave'} at 127.0.99.136:36575)
I20240503 07:59:15.462415 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 558c0eb4a4334dbe91ac60fad4e820dc from peer 3d08a06d53b14381bee6223c4e1a6027 at {username='slave'} at 127.0.99.136:36575: session id = 3d08a06d53b14381bee6223c4e1a6027-558c0eb4a4334dbe91ac60fad4e820dc
I20240503 07:59:15.463192 8579 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.463205 6560 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.463302 6712 tablet_copy_source_session.cc:205] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.463572 11449 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.463783 9288 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.464465 11812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c0eb4a4334dbe91ac60fad4e820dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.464867 11634 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.465473 8579 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.465576 6560 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.466646 11812 tablet_copy_client.cc:783] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.466779 11812 tablet_copy_client.cc:647] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.466900 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60f643dce3eb4c669b36b0714a31651b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.467490 6156 catalog_manager.cc:5484] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.469488 11812 tablet_copy_client.cc:523] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.469949 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f643dce3eb4c669b36b0714a31651b with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:15.470194 11817 tablet_replica.cc:330] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:15.470283 11817 raft_consensus.cc:2229] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.470377 11817 raft_consensus.cc:2258] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.470603 11812 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:15.470852 11817 ts_tablet_manager.cc:1875] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.472745 11817 ts_tablet_manager.cc:1888] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:15.472826 11817 log.cc:1196] T 60f643dce3eb4c669b36b0714a31651b P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/60f643dce3eb4c669b36b0714a31651b
I20240503 07:59:15.473304 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 60f643dce3eb4c669b36b0714a31651b (table rebalance_test_table_17 [id=c5d3d4c5df5243378c8176c155dc3c5b]) successfully deleted
I20240503 07:59:15.491902 11812 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.492133 11812 tablet_bootstrap.cc:492] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap complete.
I20240503 07:59:15.492298 11812 ts_tablet_manager.cc:1367] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.001s
I20240503 07:59:15.492695 11812 raft_consensus.cc:348] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:15.492830 11812 raft_consensus.cc:729] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: LEARNER
I20240503 07:59:15.492923 11812 consensus_queue.cc:260] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: NON_VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: true } }
I20240503 07:59:15.493191 11812 ts_tablet_manager.cc:1398] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:15.493609 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 3d08a06d53b14381bee6223c4e1a6027-558c0eb4a4334dbe91ac60fad4e820dc received from {username='slave'} at 127.0.99.136:36575
I20240503 07:59:15.493682 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 3d08a06d53b14381bee6223c4e1a6027-558c0eb4a4334dbe91ac60fad4e820dc on tablet 558c0eb4a4334dbe91ac60fad4e820dc with peer 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:15.495748 6428 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47596
I20240503 07:59:15.495858 6428 raft_consensus.cc:595] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:15.515241 6149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:15.515383 6149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:15.517289 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 02b087008d24472ba3e5e644c2bba6a8
I20240503 07:59:15.517740 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:15.517825 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:15.517956 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b087008d24472ba3e5e644c2bba6a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:15 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:15.518014 11864 tablet_replica.cc:330] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:15.518129 11864 raft_consensus.cc:2229] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:15.518247 11865 tablet_replica.cc:330] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:15.518292 11866 tablet_replica.cc:330] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:15.518335 11864 raft_consensus.cc:2258] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.518429 11865 raft_consensus.cc:2229] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.518455 11866 raft_consensus.cc:2229] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.518594 11865 raft_consensus.cc:2258] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.518906 11864 ts_tablet_manager.cc:1875] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.519116 11865 ts_tablet_manager.cc:1875] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.518621 11866 raft_consensus.cc:2258] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.519946 11866 ts_tablet_manager.cc:1875] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:15.521553 11864 ts_tablet_manager.cc:1888] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.521718 11865 ts_tablet_manager.cc:1888] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.521790 11865 log.cc:1196] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/02b087008d24472ba3e5e644c2bba6a8
I20240503 07:59:15.521661 11864 log.cc:1196] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/02b087008d24472ba3e5e644c2bba6a8
I20240503 07:59:15.522061 11865 ts_tablet_manager.cc:1909] T 02b087008d24472ba3e5e644c2bba6a8 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:15.522293 11864 ts_tablet_manager.cc:1909] T 02b087008d24472ba3e5e644c2bba6a8 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:15.522488 6691 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:15.522652 11866 ts_tablet_manager.cc:1888] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:15.522668 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table_new_name [id=af669f53d59f4eb4bdc4b498325400dd]) successfully deleted
I20240503 07:59:15.522861 11866 log.cc:1196] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/02b087008d24472ba3e5e644c2bba6a8
I20240503 07:59:15.523267 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table_new_name [id=af669f53d59f4eb4bdc4b498325400dd]) successfully deleted
I20240503 07:59:15.523710 11866 ts_tablet_manager.cc:1909] T 02b087008d24472ba3e5e644c2bba6a8 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:15.523748 6428 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.523835 8977 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.524137 9347 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.524291 10254 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:15.524492 6625 consensus_peers.cc:485] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8861b52c306e43d8b0615202cf7d8e35. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.524801 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 02b087008d24472ba3e5e644c2bba6a8 (table rebalancer_dynamic_table_new_name [id=af669f53d59f4eb4bdc4b498325400dd]) successfully deleted
I20240503 07:59:15.525830 9968 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.526278 6428 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.527598 8977 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.528213 6149 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.555404 10254 raft_consensus.cc:1050] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:15.555778 10254 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:15.556970 8579 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.556958 8842 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.556989 6560 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:15.557301 9968 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.557489 8296 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.558300 9347 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:15.560267 8296 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.560612 8579 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.562652 6560 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.563280 6155 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.565865 8842 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.567085 6691 consensus_queue.cc:237] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:15.568259 8842 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:15.568629 11905 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:15.569136 8579 raft_consensus.cc:1261] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:15.569518 11905 consensus_queue.cc:1035] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:15.570644 8296 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.571204 8842 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.571970 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39b01318dc1543cba2cc991e9db91c28 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.572842 6156 catalog_manager.cc:5484] T 39b01318dc1543cba2cc991e9db91c28 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.573176 8579 raft_consensus.cc:2932] T 39b01318dc1543cba2cc991e9db91c28 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.575944 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b01318dc1543cba2cc991e9db91c28 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:15.576329 11912 tablet_replica.cc:330] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:15.576540 11912 raft_consensus.cc:2229] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.576740 11912 raft_consensus.cc:2258] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.577358 11912 ts_tablet_manager.cc:1875] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.579540 11912 ts_tablet_manager.cc:1888] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:15.579654 11912 log.cc:1196] T 39b01318dc1543cba2cc991e9db91c28 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/39b01318dc1543cba2cc991e9db91c28
I20240503 07:59:15.580160 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 39b01318dc1543cba2cc991e9db91c28 (table rebalance_test_table_15 [id=eda5136a1c36474ca09f041825ad0c43]) successfully deleted
I20240503 07:59:15.583501 11918 ts_tablet_manager.cc:902] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:15.583954 11918 tablet_copy_client.cc:308] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:15.584807 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 551f25dcee034b7894498e3c748b0f0e from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:50419)
I20240503 07:59:15.584910 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 551f25dcee034b7894498e3c748b0f0e from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:50419: session id = d1f36d57661b4429a4be8d10b58d8ff7-551f25dcee034b7894498e3c748b0f0e
I20240503 07:59:15.585812 6318 tablet_copy_source_session.cc:205] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.586578 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551f25dcee034b7894498e3c748b0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.588498 11918 tablet_copy_client.cc:783] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.588639 11918 tablet_copy_client.cc:647] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.589638 11918 tablet_copy_client.cc:523] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.590842 11918 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:15.594272 11918 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.594514 11918 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:15.594648 11918 ts_tablet_manager.cc:1367] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:15.595011 11918 raft_consensus.cc:348] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.595124 11918 raft_consensus.cc:729] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:15.595214 11918 consensus_queue.cc:260] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.595526 11918 ts_tablet_manager.cc:1398] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.595940 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-551f25dcee034b7894498e3c748b0f0e received from {username='slave'} at 127.0.99.130:50419
I20240503 07:59:15.596014 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-551f25dcee034b7894498e3c748b0f0e on tablet 551f25dcee034b7894498e3c748b0f0e with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:15.619019 6149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:15.622169 6428 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47632
I20240503 07:59:15.622301 6428 raft_consensus.cc:595] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:15.624977 8824 tablet_service.cc:1467] Processing CreateTablet for tablet f11cc506153f40799edd44eb72576c77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]), partition=
I20240503 07:59:15.625237 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11cc506153f40799edd44eb72576c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.625409 6277 tablet_service.cc:1467] Processing CreateTablet for tablet f11cc506153f40799edd44eb72576c77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]), partition=
I20240503 07:59:15.625598 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11cc506153f40799edd44eb72576c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.627229 11570 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.627628 8560 tablet_service.cc:1467] Processing CreateTablet for tablet f11cc506153f40799edd44eb72576c77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]), partition=
I20240503 07:59:15.627839 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11cc506153f40799edd44eb72576c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.628016 11570 tablet_bootstrap.cc:654] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.629319 11570 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:15.629417 11570 ts_tablet_manager.cc:1367] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.629734 11056 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.629791 11570 raft_consensus.cc:348] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.629879 11570 raft_consensus.cc:374] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.629904 11570 raft_consensus.cc:729] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.630010 11570 consensus_queue.cc:260] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.630249 11570 ts_tablet_manager.cc:1398] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.630497 11056 tablet_bootstrap.cc:654] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.632714 11056 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.632860 11056 ts_tablet_manager.cc:1367] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:15.633235 11954 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:15.633265 11056 raft_consensus.cc:348] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.633348 11056 raft_consensus.cc:374] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.633375 11056 raft_consensus.cc:729] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.633500 11056 consensus_queue.cc:260] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.633755 11056 ts_tablet_manager.cc:1398] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.635460 11954 tablet_bootstrap.cc:654] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.636436 11956 ts_tablet_manager.cc:902] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:15.638121 11954 tablet_bootstrap.cc:492] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:15.638208 11954 ts_tablet_manager.cc:1367] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:15.638578 11954 raft_consensus.cc:348] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.638682 11954 raft_consensus.cc:374] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.639040 11956 tablet_copy_client.cc:308] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:15.639089 11954 raft_consensus.cc:729] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.639286 11954 consensus_queue.cc:260] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.639531 11954 ts_tablet_manager.cc:1398] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.639803 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet c8cf3d6a3ace46769854ed61f340fb25 from peer c167a0a433d34b74affed13cbf96e302 ({username='slave'} at 127.0.99.129:52937)
I20240503 07:59:15.639892 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet c8cf3d6a3ace46769854ed61f340fb25 from peer c167a0a433d34b74affed13cbf96e302 at {username='slave'} at 127.0.99.129:52937: session id = c167a0a433d34b74affed13cbf96e302-c8cf3d6a3ace46769854ed61f340fb25
I20240503 07:59:15.640609 8732 tablet_copy_source_session.cc:205] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.641695 11956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8cf3d6a3ace46769854ed61f340fb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.643759 11956 tablet_copy_client.cc:783] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.643904 11956 tablet_copy_client.cc:647] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.644148 11371 raft_consensus.cc:1050] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:15.644551 11371 consensus_queue.cc:237] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:15.645124 11956 tablet_copy_client.cc:523] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.646533 11956 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:15.646595 6691 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.646593 8712 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.646899 6428 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.647405 11046 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.647567 11370 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.647704 9803 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.650823 11956 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.651033 11956 tablet_bootstrap.cc:492] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Bootstrap complete.
I20240503 07:59:15.651166 11956 ts_tablet_manager.cc:1367] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:15.651454 9803 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.651613 11956 raft_consensus.cc:348] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:15.651690 11956 raft_consensus.cc:729] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: LEARNER
I20240503 07:59:15.651767 11956 consensus_queue.cc:260] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:15.651997 11956 ts_tablet_manager.cc:1398] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.653041 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session c167a0a433d34b74affed13cbf96e302-c8cf3d6a3ace46769854ed61f340fb25 received from {username='slave'} at 127.0.99.129:52937
I20240503 07:59:15.653098 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session c167a0a433d34b74affed13cbf96e302-c8cf3d6a3ace46769854ed61f340fb25 on tablet c8cf3d6a3ace46769854ed61f340fb25 with peer c167a0a433d34b74affed13cbf96e302
I20240503 07:59:15.653034 6428 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.653026 6691 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.654706 8712 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.656189 6149 catalog_manager.cc:5484] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.659197 6298 consensus_queue.cc:237] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:15.660207 8712 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.660269 6428 raft_consensus.cc:1261] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.660596 11370 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.660756 11046 consensus_queue.cc:1035] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.662251 11046 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.663122 8712 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.663892 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bfd4259777e5492c904e6d65a36d92ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.664175 6428 raft_consensus.cc:2932] T bfd4259777e5492c904e6d65a36d92ed P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.665900 6155 catalog_manager.cc:5484] T bfd4259777e5492c904e6d65a36d92ed P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:15.667856 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd4259777e5492c904e6d65a36d92ed with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:15.668059 11817 tablet_replica.cc:330] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:15.668191 11817 raft_consensus.cc:2229] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.668336 11817 raft_consensus.cc:2258] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.669034 11817 ts_tablet_manager.cc:1875] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.670835 11817 ts_tablet_manager.cc:1888] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:15.670883 11817 log.cc:1196] T bfd4259777e5492c904e6d65a36d92ed P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/bfd4259777e5492c904e6d65a36d92ed
I20240503 07:59:15.671249 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet bfd4259777e5492c904e6d65a36d92ed (table rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]) successfully deleted
I20240503 07:59:15.673342 6428 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:15.674628 8977 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.674687 6560 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:15.674742 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a10085c34be4d31aae9f56a2ec1ba23. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.674968 11449 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.675148 11427 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.676676 11449 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.677115 8977 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.677816 6560 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:15.679844 6155 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.681402 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.684720 11521 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.720265 11589 raft_consensus.cc:482] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.720366 11589 raft_consensus.cc:504] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.720698 11589 leader_election.cc:288] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:15.720965 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11cc506153f40799edd44eb72576c77" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:15.720965 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11cc506153f40799edd44eb72576c77" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:15.721088 6298 raft_consensus.cc:2454] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:15.721088 8842 raft_consensus.cc:2454] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:15.721423 8514 leader_election.cc:302] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.721690 11589 raft_consensus.cc:2781] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.721747 11589 raft_consensus.cc:482] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.721782 11589 raft_consensus.cc:3037] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.722626 11589 raft_consensus.cc:504] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.722937 11589 leader_election.cc:288] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:15.723242 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11cc506153f40799edd44eb72576c77" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:15.723348 6298 raft_consensus.cc:3037] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.723418 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11cc506153f40799edd44eb72576c77" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:15.723505 8842 raft_consensus.cc:3037] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.724270 6298 raft_consensus.cc:2454] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:15.724280 8842 raft_consensus.cc:2454] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:15.724581 8514 leader_election.cc:302] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.724725 11589 raft_consensus.cc:2781] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.724902 11589 raft_consensus.cc:686] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:15.725031 11589 consensus_queue.cc:237] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:15.726704 6157 catalog_manager.cc:5484] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.739388 9288 raft_consensus.cc:979] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: : Instructing follower c8d71787690245e681c85a087086e7ed to start an election
I20240503 07:59:15.739482 9288 raft_consensus.cc:1067] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Signalling peer c8d71787690245e681c85a087086e7ed to start an election
I20240503 07:59:15.740216 8579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003"
dest_uuid: "c8d71787690245e681c85a087086e7ed"
from {username='slave'} at 127.0.99.130:45801
I20240503 07:59:15.740347 8579 raft_consensus.cc:482] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:15.740432 8579 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:15.741155 8579 raft_consensus.cc:504] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.741626 8579 leader_election.cc:288] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:15.741916 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:15.742018 6428 raft_consensus.cc:3032] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:15.742055 6428 raft_consensus.cc:729] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:15.742175 6428 consensus_queue.cc:260] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.742337 6428 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:15.742380 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:15.742497 6691 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:15.743201 6428 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:15.743343 6691 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:15.743623 8516 leader_election.cc:302] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:15.743781 11589 raft_consensus.cc:2781] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:15.743872 11589 raft_consensus.cc:686] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:15.743993 11589 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.745784 6157 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.748001 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5631f3e97fe04013a2a7140f44260003" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:15.748114 8977 raft_consensus.cc:3037] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:15.749055 8977 raft_consensus.cc:2454] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:15.752696 11918 ts_tablet_manager.cc:902] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:15.753109 11918 tablet_copy_client.cc:308] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:15.753558 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet e103e0e9cb16416dba7f8bbfc08ceac8 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:50419)
I20240503 07:59:15.753644 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet e103e0e9cb16416dba7f8bbfc08ceac8 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:50419: session id = d1f36d57661b4429a4be8d10b58d8ff7-e103e0e9cb16416dba7f8bbfc08ceac8
I20240503 07:59:15.754314 6318 tablet_copy_source_session.cc:205] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.755052 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e103e0e9cb16416dba7f8bbfc08ceac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.756654 11918 tablet_copy_client.cc:783] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.757270 11918 tablet_copy_client.cc:647] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.758249 11918 tablet_copy_client.cc:523] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.759338 11918 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:15.762236 11918 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.762447 11918 tablet_bootstrap.cc:492] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:15.762595 11918 ts_tablet_manager.cc:1367] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.004s
I20240503 07:59:15.763203 11918 raft_consensus.cc:348] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.763311 11918 raft_consensus.cc:729] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:15.763410 11918 consensus_queue.cc:260] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:15.763669 11918 ts_tablet_manager.cc:1398] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:15.764098 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-e103e0e9cb16416dba7f8bbfc08ceac8 received from {username='slave'} at 127.0.99.130:50419
I20240503 07:59:15.764171 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-e103e0e9cb16416dba7f8bbfc08ceac8 on tablet e103e0e9cb16416dba7f8bbfc08ceac8 with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:15.779695 12092 ts_tablet_manager.cc:902] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:15.780176 12092 tablet_copy_client.cc:308] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:15.780617 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 974c815ff8e94190a7bf28444cb3cd9f from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:40145)
I20240503 07:59:15.780702 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 974c815ff8e94190a7bf28444cb3cd9f from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:40145: session id = 9553fe3c4ccd4e91b7c176aad82fec09-974c815ff8e94190a7bf28444cb3cd9f
I20240503 07:59:15.780799 12098 ts_tablet_manager.cc:902] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:15.781248 12098 tablet_copy_client.cc:308] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39949
I20240503 07:59:15.781523 6450 tablet_copy_source_session.cc:205] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.782269 12092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974c815ff8e94190a7bf28444cb3cd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.782586 8601 tablet_copy_service.cc:140] P c8d71787690245e681c85a087086e7ed: Received BeginTabletCopySession request for tablet c5b4c776eac24f739d85e09707cd751b from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:49789)
I20240503 07:59:15.782696 8601 tablet_copy_service.cc:161] P c8d71787690245e681c85a087086e7ed: Beginning new tablet copy session on tablet c5b4c776eac24f739d85e09707cd751b from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:49789: session id = ad910f92ef9d4db8a8962cecba289f8a-c5b4c776eac24f739d85e09707cd751b
I20240503 07:59:15.783938 8601 tablet_copy_source_session.cc:205] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.784062 12092 tablet_copy_client.cc:783] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.784193 12092 tablet_copy_client.cc:647] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.784991 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4c776eac24f739d85e09707cd751b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.785502 12092 tablet_copy_client.cc:523] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.786633 12092 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:15.787127 12098 tablet_copy_client.cc:783] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.787256 12098 tablet_copy_client.cc:647] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.789093 12098 tablet_copy_client.cc:523] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.790127 12098 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:15.793115 12098 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.793325 12098 tablet_bootstrap.cc:492] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:15.793471 12098 ts_tablet_manager.cc:1367] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:15.793840 12098 raft_consensus.cc:348] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:15.793938 12098 raft_consensus.cc:729] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:15.794054 12098 consensus_queue.cc:260] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:15.794392 12098 ts_tablet_manager.cc:1398] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.794859 8601 tablet_copy_service.cc:342] P c8d71787690245e681c85a087086e7ed: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-c5b4c776eac24f739d85e09707cd751b received from {username='slave'} at 127.0.99.131:49789
I20240503 07:59:15.794943 8601 tablet_copy_service.cc:433] P c8d71787690245e681c85a087086e7ed: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-c5b4c776eac24f739d85e09707cd751b on tablet c5b4c776eac24f739d85e09707cd751b with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:15.797042 10079 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.797749 12115 ts_tablet_manager.cc:902] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:15.798187 12115 tablet_copy_client.cc:308] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:15.799957 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet c597bc1a19274f55b9d60558db35fcae from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:33533)
I20240503 07:59:15.800038 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet c597bc1a19274f55b9d60558db35fcae from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:33533: session id = 616fdda6ba714cb7be01a564de2b9db2-c597bc1a19274f55b9d60558db35fcae
I20240503 07:59:15.800837 6581 tablet_copy_source_session.cc:205] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.801684 12115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c597bc1a19274f55b9d60558db35fcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.802739 9400 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.804975 12115 tablet_copy_client.cc:783] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.805119 12115 tablet_copy_client.cc:647] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.806941 12115 tablet_copy_client.cc:523] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.808719 12115 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.811854 12092 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.812116 12092 tablet_bootstrap.cc:492] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:15.812271 12092 ts_tablet_manager.cc:1367] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240503 07:59:15.812641 12092 raft_consensus.cc:348] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:15.812752 12092 raft_consensus.cc:729] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:15.812860 12092 consensus_queue.cc:260] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:15.813333 12092 ts_tablet_manager.cc:1398] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.813869 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-974c815ff8e94190a7bf28444cb3cd9f received from {username='slave'} at 127.0.99.134:40145
I20240503 07:59:15.813935 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-974c815ff8e94190a7bf28444cb3cd9f on tablet 974c815ff8e94190a7bf28444cb3cd9f with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:15.817726 9400 raft_consensus.cc:1050] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:15.818068 9400 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.819217 6298 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.819217 8579 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.819267 8977 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.819558 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.819891 10815 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.820053 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.821835 10815 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.823385 8579 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.824647 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:15.824841 6156 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 2 to 3, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.827533 8977 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.831907 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:15.835345 8561 tablet_service.cc:1467] Processing CreateTablet for tablet 7995f12a988548088d45dc4d7a1b9aa0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.835590 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995f12a988548088d45dc4d7a1b9aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.835778 8560 tablet_service.cc:1467] Processing CreateTablet for tablet fa8b855a6985404da6bd34daa0eb9ae7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.836020 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8b855a6985404da6bd34daa0eb9ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.835963 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 5b745ba777b2476fbae78706b35125f6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.836199 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b745ba777b2476fbae78706b35125f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.837100 8824 tablet_service.cc:1467] Processing CreateTablet for tablet fa8b855a6985404da6bd34daa0eb9ae7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.837316 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8b855a6985404da6bd34daa0eb9ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.837563 11056 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.837596 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 7995f12a988548088d45dc4d7a1b9aa0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.837807 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995f12a988548088d45dc4d7a1b9aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.838372 11056 tablet_bootstrap.cc:654] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.839186 11570 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.839454 12115 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.839689 12115 tablet_bootstrap.cc:492] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:15.839828 12115 ts_tablet_manager.cc:1367] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.000s
I20240503 07:59:15.839784 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 7995f12a988548088d45dc4d7a1b9aa0 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.839921 11570 tablet_bootstrap.cc:654] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.839879 6410 tablet_service.cc:1467] Processing CreateTablet for tablet fa8b855a6985404da6bd34daa0eb9ae7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.840004 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995f12a988548088d45dc4d7a1b9aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.840111 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8b855a6985404da6bd34daa0eb9ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.840222 12115 raft_consensus.cc:348] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:15.840315 12115 raft_consensus.cc:729] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:15.840395 12115 consensus_queue.cc:260] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:15.840538 12138 ts_tablet_manager.cc:902] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:15.841012 12138 tablet_copy_client.cc:308] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:15.841203 12115 ts_tablet_manager.cc:1398] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:15.842235 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 5b745ba777b2476fbae78706b35125f6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.841948 11056 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.842450 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b745ba777b2476fbae78706b35125f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.842444 11056 ts_tablet_manager.cc:1367] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:15.842635 6298 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.842948 11056 raft_consensus.cc:348] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.843029 11056 raft_consensus.cc:374] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.843053 11056 raft_consensus.cc:729] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.843155 11056 consensus_queue.cc:260] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.843374 11056 ts_tablet_manager.cc:1398] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:15.846163 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 82de209205b342cfaac10506fdac9303 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:54869)
I20240503 07:59:15.846191 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-c597bc1a19274f55b9d60558db35fcae received from {username='slave'} at 127.0.99.135:33533
I20240503 07:59:15.846275 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-c597bc1a19274f55b9d60558db35fcae on tablet c597bc1a19274f55b9d60558db35fcae with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:15.846261 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 82de209205b342cfaac10506fdac9303 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:54869: session id = c8d71787690245e681c85a087086e7ed-82de209205b342cfaac10506fdac9303
I20240503 07:59:15.847160 6712 tablet_copy_source_session.cc:205] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.847723 8712 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:15.848419 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82de209205b342cfaac10506fdac9303. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.848922 8579 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:15.849416 11570 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:15.849490 11570 ts_tablet_manager.cc:1367] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:15.849510 9963 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.849869 11570 raft_consensus.cc:348] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.849957 11570 raft_consensus.cc:374] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.849987 11570 raft_consensus.cc:729] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.850085 11570 consensus_queue.cc:260] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.850103 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.850318 11570 ts_tablet_manager.cc:1398] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.851295 11056 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.852069 11056 tablet_bootstrap.cc:654] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.852325 12138 tablet_copy_client.cc:783] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.852455 12138 tablet_copy_client.cc:647] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.854156 12138 tablet_copy_client.cc:523] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:15.854617 11648 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.855999 12138 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.856046 6298 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:15.856357 10079 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.857702 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 5b745ba777b2476fbae78706b35125f6 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:15.857916 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b745ba777b2476fbae78706b35125f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.858489 8843 raft_consensus.cc:1261] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.858755 6298 raft_consensus.cc:1261] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:15.859146 10079 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.859201 12035 consensus_queue.cc:1035] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.859449 12035 consensus_queue.cc:1035] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:15.859644 11570 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:15.860353 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f635e30fb6974fa1a79d8a46eecc3d5e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.860536 11570 tablet_bootstrap.cc:654] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.861435 6149 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.861670 6155 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]). Expected version 1 got 0
I20240503 07:59:15.861717 8579 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.862771 6296 raft_consensus.cc:2932] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:15.865967 6149 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]). Expected version 1 got 0
I20240503 07:59:15.866670 11056 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.866750 11056 ts_tablet_manager.cc:1367] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:15.866904 11570 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:15.866986 11570 ts_tablet_manager.cc:1367] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:15.867123 11056 raft_consensus.cc:348] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.867210 11056 raft_consensus.cc:374] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.867239 11056 raft_consensus.cc:729] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.867333 11056 consensus_queue.cc:260] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.867367 11570 raft_consensus.cc:348] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.867450 11570 raft_consensus.cc:374] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.867480 11570 raft_consensus.cc:729] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.867596 11056 ts_tablet_manager.cc:1398] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.867578 11570 consensus_queue.cc:260] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.867728 11056 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:15.867820 11570 ts_tablet_manager.cc:1398] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.868489 11056 tablet_bootstrap.cc:654] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.870126 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet f635e30fb6974fa1a79d8a46eecc3d5e with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:15.870890 12155 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:15.871793 12155 tablet_bootstrap.cc:654] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.873912 12140 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:15.874123 12146 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:15.875430 11602 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.875662 11602 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:15.874794 12161 tablet_replica.cc:330] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:15.875834 12161 raft_consensus.cc:2229] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.876911 12161 raft_consensus.cc:2258] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.877491 6149 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]). Expected version 1 got 0
I20240503 07:59:15.877969 6153 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]). Expected version 1 got 0
I20240503 07:59:15.878834 11056 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:15.878916 11056 ts_tablet_manager.cc:1367] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:15.879361 11056 raft_consensus.cc:348] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.879446 11056 raft_consensus.cc:374] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.879475 11056 raft_consensus.cc:729] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.879572 11056 consensus_queue.cc:260] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.879812 11056 ts_tablet_manager.cc:1398] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.880951 12161 ts_tablet_manager.cc:1875] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.881667 12146 tablet_bootstrap.cc:654] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.881861 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350] alter complete (version 1)
I20240503 07:59:15.882428 12140 tablet_bootstrap.cc:654] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.883626 8296 raft_consensus.cc:1050] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:15.883899 6153 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350]). Expected version 1 got 0
I20240503 07:59:15.884035 8296 consensus_queue.cc:237] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:15.885524 6560 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.885634 6298 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.886092 8843 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:15.886626 10254 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.887221 11905 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.887514 11905 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:15.888017 12161 ts_tablet_manager.cc:1888] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:15.888516 12165 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.888886 12165 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:15.888995 12161 log.cc:1196] T f635e30fb6974fa1a79d8a46eecc3d5e P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f635e30fb6974fa1a79d8a46eecc3d5e
I20240503 07:59:15.889305 12155 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:15.889468 12155 ts_tablet_manager.cc:1367] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:15.889731 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet f635e30fb6974fa1a79d8a46eecc3d5e (table rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]) successfully deleted
I20240503 07:59:15.889989 12155 raft_consensus.cc:348] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.890143 12155 raft_consensus.cc:374] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.890225 12155 raft_consensus.cc:729] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.890367 12155 consensus_queue.cc:260] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.891160 12155 ts_tablet_manager.cc:1398] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.893922 12171 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:15.894177 12171 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:15.894469 12140 tablet_bootstrap.cc:492] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:15.894564 12140 ts_tablet_manager.cc:1367] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:15.896085 12140 raft_consensus.cc:348] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.896193 12140 raft_consensus.cc:374] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.896265 12140 raft_consensus.cc:729] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.896396 12140 consensus_queue.cc:260] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:15.896661 12140 ts_tablet_manager.cc:1398] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:15.897441 12146 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:15.897534 12146 ts_tablet_manager.cc:1367] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:15.898248 12146 raft_consensus.cc:348] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.898355 12146 raft_consensus.cc:374] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.898387 12146 raft_consensus.cc:729] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.898607 12146 consensus_queue.cc:260] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.898895 12146 ts_tablet_manager.cc:1398] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:15.899026 12146 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:15.900199 12146 tablet_bootstrap.cc:654] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:15.901697 11905 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.902063 6560 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.902570 12146 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:15.902639 12146 ts_tablet_manager.cc:1367] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:15.903512 6298 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.904636 6155 catalog_manager.cc:5484] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.903084 12146 raft_consensus.cc:348] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.905531 12146 raft_consensus.cc:374] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:15.905573 12146 raft_consensus.cc:729] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:15.905874 12146 consensus_queue.cc:260] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.906134 12146 ts_tablet_manager.cc:1398] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:15.908252 8843 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.910954 6691 consensus_queue.cc:237] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:15.911870 8843 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.912123 6560 raft_consensus.cc:1261] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:15.912351 9347 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.912943 9347 consensus_queue.cc:1035] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:15.914044 9347 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.915370 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d3f03610866454595f4b6b87b7cd9cb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.915629 6560 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.915659 8842 raft_consensus.cc:2932] T 2d3f03610866454595f4b6b87b7cd9cb P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:15.916347 6153 catalog_manager.cc:5484] T 2d3f03610866454595f4b6b87b7cd9cb P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.918601 11905 raft_consensus.cc:1050] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:15.918927 11905 consensus_queue.cc:237] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:15.920430 8977 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.920542 12138 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:15.920630 6428 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.920804 12138 tablet_bootstrap.cc:492] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:15.920989 9968 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.921103 12138 ts_tablet_manager.cc:1367] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.066s user 0.017s sys 0.009s
I20240503 07:59:15.921213 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3f03610866454595f4b6b87b7cd9cb with delete_type TABLET_DATA_TOMBSTONED (TS c167a0a433d34b74affed13cbf96e302 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:15.921811 12138 raft_consensus.cc:348] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:15.921908 12138 raft_consensus.cc:729] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:15.921957 12190 tablet_replica.cc:330] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:15.921923 8296 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.921991 12138 consensus_queue.cc:260] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:15.922097 12190 raft_consensus.cc:2229] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.922267 12190 raft_consensus.cc:2258] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.922326 12138 ts_tablet_manager.cc:1398] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:15.923547 12190 ts_tablet_manager.cc:1875] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.923856 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-82de209205b342cfaac10506fdac9303 received from {username='slave'} at 127.0.99.133:54869
I20240503 07:59:15.923921 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session c8d71787690245e681c85a087086e7ed-82de209205b342cfaac10506fdac9303 on tablet 82de209205b342cfaac10506fdac9303 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:15.924916 8842 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.925194 9968 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.926064 12190 ts_tablet_manager.cc:1888] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:15.926257 12190 log.cc:1196] T 2d3f03610866454595f4b6b87b7cd9cb P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2d3f03610866454595f4b6b87b7cd9cb
I20240503 07:59:15.925271 11905 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.926903 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 2d3f03610866454595f4b6b87b7cd9cb (table rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]) successfully deleted
I20240503 07:59:15.927035 6428 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.927457 8842 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.929795 6155 catalog_manager.cc:5484] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.930188 8977 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.932952 6691 consensus_queue.cc:237] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:15.934244 6428 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:15.934300 8977 raft_consensus.cc:1261] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:15.934620 9968 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.934845 9968 consensus_queue.cc:1035] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:15.936136 9968 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.936515 8977 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.937809 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4e4ca6638764fb49f3afcab7688e2b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:15.937733 6428 raft_consensus.cc:2932] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.939872 6153 catalog_manager.cc:5484] T f4e4ca6638764fb49f3afcab7688e2b3 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:15.941946 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e4ca6638764fb49f3afcab7688e2b3 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:15.942343 12204 tablet_replica.cc:330] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:15.942497 12204 raft_consensus.cc:2229] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:15.942654 12204 raft_consensus.cc:2258] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:15.943257 12204 ts_tablet_manager.cc:1875] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:15.945372 12204 ts_tablet_manager.cc:1888] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:15.945452 12204 log.cc:1196] T f4e4ca6638764fb49f3afcab7688e2b3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f4e4ca6638764fb49f3afcab7688e2b3
I20240503 07:59:15.945922 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet f4e4ca6638764fb49f3afcab7688e2b3 (table rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]) successfully deleted
I20240503 07:59:15.962483 10916 raft_consensus.cc:482] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.962574 10916 raft_consensus.cc:504] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.962888 10916 leader_election.cc:288] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:15.963156 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b745ba777b2476fbae78706b35125f6" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:15.963193 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b745ba777b2476fbae78706b35125f6" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:15.963274 6428 raft_consensus.cc:2454] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:15.963289 8579 raft_consensus.cc:2454] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:15.963533 8779 leader_election.cc:302] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:15.963714 10916 raft_consensus.cc:2781] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.963802 10916 raft_consensus.cc:482] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.963840 10916 raft_consensus.cc:3037] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.964578 10916 raft_consensus.cc:504] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.964915 10916 leader_election.cc:288] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:15.965207 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b745ba777b2476fbae78706b35125f6" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:15.965272 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b745ba777b2476fbae78706b35125f6" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:15.965315 6428 raft_consensus.cc:3037] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.965380 8579 raft_consensus.cc:3037] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.966148 6428 raft_consensus.cc:2454] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:15.966148 8579 raft_consensus.cc:2454] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:15.966470 8779 leader_election.cc:302] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:15.966606 10916 raft_consensus.cc:2781] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.966676 10916 raft_consensus.cc:686] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:15.966784 10916 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:15.969262 6155 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:15.978013 6298 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:15.978631 11589 raft_consensus.cc:482] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:15.978717 11589 raft_consensus.cc:504] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.979043 11589 leader_election.cc:288] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:15.979357 6691 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.979369 8842 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:15.979390 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8b855a6985404da6bd34daa0eb9ae7" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:15.979471 6428 raft_consensus.cc:2454] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:15.979585 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8b855a6985404da6bd34daa0eb9ae7" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:15.979691 8843 raft_consensus.cc:2454] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:15.979727 8517 leader_election.cc:302] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:15.979890 10239 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.980163 9803 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:15.980247 11589 raft_consensus.cc:2781] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:15.980295 11589 raft_consensus.cc:482] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:15.980329 11589 raft_consensus.cc:3037] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:15.980391 6232 consensus_peers.cc:485] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4308bc87be24f28b6566ce76cae6e50. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:15.981272 11589 raft_consensus.cc:504] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.981578 11589 leader_election.cc:288] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:15.981719 10239 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.982048 6428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8b855a6985404da6bd34daa0eb9ae7" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:15.982055 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8b855a6985404da6bd34daa0eb9ae7" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:15.982134 6428 raft_consensus.cc:3037] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.982134 8842 raft_consensus.cc:3037] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:15.982307 11508 raft_consensus.cc:1050] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:15.982800 11508 consensus_queue.cc:237] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:15.982906 8842 raft_consensus.cc:2454] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:15.982921 6428 raft_consensus.cc:2454] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:15.983191 8516 leader_election.cc:302] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:15.983705 11589 raft_consensus.cc:2781] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:15.983790 11589 raft_consensus.cc:686] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:15.983808 12098 ts_tablet_manager.cc:902] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:15.983908 11589 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:15.984217 12098 tablet_copy_client.cc:308] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:15.985060 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet 09866b93f75545a19c9451556254da48 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:38091)
I20240503 07:59:15.985183 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet 09866b93f75545a19c9451556254da48 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:38091: session id = ad910f92ef9d4db8a8962cecba289f8a-09866b93f75545a19c9451556254da48
I20240503 07:59:15.985438 6155 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.986107 8732 tablet_copy_source_session.cc:205] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:15.986205 8977 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.986927 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09866b93f75545a19c9451556254da48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:15.987195 6151 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:15.988196 12241 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.988456 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:15.989988 6560 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.990797 8579 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:15.991279 12240 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.991652 12240 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:15.991783 6691 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.994403 8843 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } } }
I20240503 07:59:15.995944 12098 tablet_copy_client.cc:783] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:15.996085 12098 tablet_copy_client.cc:647] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:15.998895 11602 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:15.999161 11602 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:15.999938 10916 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.000473 12171 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.000747 12171 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:16.001698 8977 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.003476 6560 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.005784 12165 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.006026 12165 tablet.cc:1620] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:16.006120 6153 catalog_manager.cc:5484] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.006340 12098 tablet_copy_client.cc:523] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.008062 12098 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:16.008736 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=764177922b50411d8c7f7f5cc753f350] alter complete (version 2)
I20240503 07:59:16.011549 8579 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.015914 8843 consensus_queue.cc:237] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:16.016319 12098 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.016528 12098 tablet_bootstrap.cc:492] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:16.016656 12098 ts_tablet_manager.cc:1367] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 07:59:16.017001 8579 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.017038 6560 raft_consensus.cc:1261] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.017066 12098 raft_consensus.cc:348] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:16.017160 12098 raft_consensus.cc:729] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:16.017261 12098 consensus_queue.cc:260] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:16.017395 12240 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.017529 12098 ts_tablet_manager.cc:1398] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.017699 12240 consensus_queue.cc:1035] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.017982 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-09866b93f75545a19c9451556254da48 received from {username='slave'} at 127.0.99.131:38091
I20240503 07:59:16.018049 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-09866b93f75545a19c9451556254da48 on tablet 09866b93f75545a19c9451556254da48 with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:16.020221 11508 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.021572 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5060df68b95e48a8a849e77ba926e608 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.022836 6560 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.023488 8579 raft_consensus.cc:2932] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.026147 6155 catalog_manager.cc:5484] T 5060df68b95e48a8a849e77ba926e608 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.028707 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 5060df68b95e48a8a849e77ba926e608 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:16.028936 12161 tablet_replica.cc:330] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:16.029057 12161 raft_consensus.cc:2229] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.029173 12161 raft_consensus.cc:2258] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.029701 12161 ts_tablet_manager.cc:1875] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.031456 12161 ts_tablet_manager.cc:1888] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:16.031509 12161 log.cc:1196] T 5060df68b95e48a8a849e77ba926e608 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/5060df68b95e48a8a849e77ba926e608
I20240503 07:59:16.032073 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 5060df68b95e48a8a849e77ba926e608 (table rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]) successfully deleted
I20240503 07:59:16.063210 11589 raft_consensus.cc:482] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.063326 11589 raft_consensus.cc:504] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.063709 11589 leader_election.cc:288] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.064298 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995f12a988548088d45dc4d7a1b9aa0" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:16.064432 8712 raft_consensus.cc:2454] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:16.064693 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995f12a988548088d45dc4d7a1b9aa0" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.064826 8517 leader_election.cc:302] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:16.064877 8977 raft_consensus.cc:2454] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:16.065858 11589 raft_consensus.cc:2781] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.065922 11589 raft_consensus.cc:482] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.065961 11589 raft_consensus.cc:3037] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.066920 11589 raft_consensus.cc:504] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.067273 11589 leader_election.cc:288] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.067626 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995f12a988548088d45dc4d7a1b9aa0" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.067729 8977 raft_consensus.cc:3037] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.068074 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995f12a988548088d45dc4d7a1b9aa0" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:16.068182 8712 raft_consensus.cc:3037] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.068614 8977 raft_consensus.cc:2454] T 7995f12a988548088d45dc4d7a1b9aa0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:16.068944 8712 raft_consensus.cc:2454] T 7995f12a988548088d45dc4d7a1b9aa0 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:16.068974 8516 leader_election.cc:302] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:16.069142 11589 raft_consensus.cc:2781] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.069224 11589 raft_consensus.cc:686] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:16.069378 11589 consensus_queue.cc:237] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.071451 6149 catalog_manager.cc:5484] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.082262 8977 raft_consensus.cc:1203] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:16.098802 12092 ts_tablet_manager.cc:902] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:16.099191 12092 tablet_copy_client.cc:308] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:16.099682 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 2a10085c34be4d31aae9f56a2ec1ba23 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:40145)
I20240503 07:59:16.099797 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 2a10085c34be4d31aae9f56a2ec1ba23 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:40145: session id = 9553fe3c4ccd4e91b7c176aad82fec09-2a10085c34be4d31aae9f56a2ec1ba23
I20240503 07:59:16.100663 6450 tablet_copy_source_session.cc:205] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:16.101516 12092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a10085c34be4d31aae9f56a2ec1ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.102376 12138 ts_tablet_manager.cc:902] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:16.102830 12138 tablet_copy_client.cc:308] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:16.103308 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 8861b52c306e43d8b0615202cf7d8e35 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:54869)
I20240503 07:59:16.103415 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 8861b52c306e43d8b0615202cf7d8e35 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:54869: session id = c8d71787690245e681c85a087086e7ed-8861b52c306e43d8b0615202cf7d8e35
I20240503 07:59:16.103575 12092 tablet_copy_client.cc:783] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:16.103698 12092 tablet_copy_client.cc:647] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:16.104207 6712 tablet_copy_source_session.cc:205] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:16.104681 12092 tablet_copy_client.cc:523] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.104970 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8861b52c306e43d8b0615202cf7d8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.105641 12092 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:16.106685 12138 tablet_copy_client.cc:783] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:16.106827 12138 tablet_copy_client.cc:647] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:16.108496 12138 tablet_copy_client.cc:523] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.109908 12138 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:16.113571 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:16.119899 11602 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.121416 12165 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.124472 12171 tablet.cc:1722] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.126312 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=764177922b50411d8c7f7f5cc753f350] alter complete (version 3)
I20240503 07:59:16.130955 6560 consensus_queue.cc:237] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:16.132330 6298 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.132404 8842 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:16.132808 6497 consensus_peers.cc:485] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afeff5ce33bc4205a8599bbb66b24672. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.132810 11755 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:16.133220 11648 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:16.134944 11755 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.135450 6298 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.138005 6155 catalog_manager.cc:5484] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.140296 8842 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.140966 12092 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.141252 12092 tablet_bootstrap.cc:492] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:16.141448 12092 ts_tablet_manager.cc:1367] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.002s
I20240503 07:59:16.141906 12138 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.141944 12092 raft_consensus.cc:348] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:16.142099 12092 raft_consensus.cc:729] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:16.142177 12138 tablet_bootstrap.cc:492] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:16.142195 12092 consensus_queue.cc:260] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:16.142331 12138 ts_tablet_manager.cc:1367] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.000s
I20240503 07:59:16.142732 12138 raft_consensus.cc:348] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:16.142891 12138 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:16.143034 12138 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:16.143370 12138 ts_tablet_manager.cc:1398] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.144744 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-8861b52c306e43d8b0615202cf7d8e35 received from {username='slave'} at 127.0.99.133:54869
I20240503 07:59:16.144872 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session c8d71787690245e681c85a087086e7ed-8861b52c306e43d8b0615202cf7d8e35 on tablet 8861b52c306e43d8b0615202cf7d8e35 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:16.145457 12092 ts_tablet_manager.cc:1398] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:16.146333 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-2a10085c34be4d31aae9f56a2ec1ba23 received from {username='slave'} at 127.0.99.134:40145
I20240503 07:59:16.146420 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-2a10085c34be4d31aae9f56a2ec1ba23 on tablet 2a10085c34be4d31aae9f56a2ec1ba23 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:16.151343 8712 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
W20240503 07:59:16.152874 8645 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a833fb4980c04673a335d020a3ad3dd3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.152997 8842 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.153143 8977 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.153838 6428 raft_consensus.cc:1203] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.153743 10815 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:16.154742 10815 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:16.157416 10079 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:16.159977 6155 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.160907 8842 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:16.160935 8977 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:16.168680 6428 raft_consensus.cc:1203] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.176642 8977 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:16.177151 11589 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.228519 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.228667 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.230690 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet f11cc506153f40799edd44eb72576c77
I20240503 07:59:16.231297 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet f11cc506153f40799edd44eb72576c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:16.231408 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet f11cc506153f40799edd44eb72576c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:16.231462 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet f11cc506153f40799edd44eb72576c77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:16.231534 12204 tablet_replica.cc:330] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:16.231685 12204 raft_consensus.cc:2229] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.231806 12204 raft_consensus.cc:2258] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.231972 12190 tablet_replica.cc:330] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:16.232084 12190 raft_consensus.cc:2229] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.232168 12190 raft_consensus.cc:2258] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.232314 12204 ts_tablet_manager.cc:1875] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.232975 12190 ts_tablet_manager.cc:1875] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.234129 12204 ts_tablet_manager.cc:1888] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.234195 12204 log.cc:1196] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f11cc506153f40799edd44eb72576c77
I20240503 07:59:16.234397 12204 ts_tablet_manager.cc:1909] T f11cc506153f40799edd44eb72576c77 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:16.234854 12190 ts_tablet_manager.cc:1888] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.234927 12190 log.cc:1196] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f11cc506153f40799edd44eb72576c77
I20240503 07:59:16.235005 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table_new_name [id=764177922b50411d8c7f7f5cc753f350]) successfully deleted
I20240503 07:59:16.235157 12190 ts_tablet_manager.cc:1909] T f11cc506153f40799edd44eb72576c77 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:16.235690 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table_new_name [id=764177922b50411d8c7f7f5cc753f350]) successfully deleted
I20240503 07:59:16.235939 12420 tablet_replica.cc:330] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:16.236059 12420 raft_consensus.cc:2229] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:16.236218 12420 raft_consensus.cc:2258] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.236666 12420 ts_tablet_manager.cc:1875] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.237614 6298 raft_consensus.cc:1203] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.241753 12420 ts_tablet_manager.cc:1888] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.241822 12420 log.cc:1196] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f11cc506153f40799edd44eb72576c77
I20240503 07:59:16.242033 12420 ts_tablet_manager.cc:1909] T f11cc506153f40799edd44eb72576c77 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:16.242570 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet f11cc506153f40799edd44eb72576c77 (table rebalancer_dynamic_table_new_name [id=764177922b50411d8c7f7f5cc753f350]) successfully deleted
I20240503 07:59:16.252542 6560 raft_consensus.cc:1203] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.261277 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:16.261757 12426 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.264403 6428 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:16.264744 11589 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.268563 8579 consensus_queue.cc:237] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:16.269536 8977 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:16.269574 6691 raft_consensus.cc:1261] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:16.269884 12035 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:16.269990 11589 consensus_queue.cc:1035] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:16.271301 11589 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.271607 8977 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.273061 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5631f3e97fe04013a2a7140f44260003 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.273283 6691 raft_consensus.cc:2932] T 5631f3e97fe04013a2a7140f44260003 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.273533 6151 catalog_manager.cc:5484] T 5631f3e97fe04013a2a7140f44260003 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 8 to 10, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.275614 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 5631f3e97fe04013a2a7140f44260003 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:16.275954 12438 tablet_replica.cc:330] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:16.276291 12438 raft_consensus.cc:2229] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.276399 12438 raft_consensus.cc:2258] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.276984 12438 ts_tablet_manager.cc:1875] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.277874 8842 raft_consensus.cc:1203] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:16.279110 12438 ts_tablet_manager.cc:1888] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 07:59:16.279170 12438 log.cc:1196] T 5631f3e97fe04013a2a7140f44260003 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5631f3e97fe04013a2a7140f44260003
I20240503 07:59:16.279568 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 5631f3e97fe04013a2a7140f44260003 (table rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]) successfully deleted
I20240503 07:59:16.304430 8712 raft_consensus.cc:1203] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:16.333472 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:16.338194 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]), partition=
I20240503 07:59:16.338271 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]), partition=
I20240503 07:59:16.338488 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a5be24d24419aa9388572b0bfd97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.338548 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a5be24d24419aa9388572b0bfd97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.340086 12140 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:16.340235 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]), partition=
I20240503 07:59:16.340262 11056 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:16.340445 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4a5be24d24419aa9388572b0bfd97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.341075 12140 tablet_bootstrap.cc:654] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.341367 11056 tablet_bootstrap.cc:654] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.342262 12140 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:16.342350 12155 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:16.342351 12140 ts_tablet_manager.cc:1367] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:16.342566 11056 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:16.342636 11056 ts_tablet_manager.cc:1367] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:16.342813 12140 raft_consensus.cc:348] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.342926 12140 raft_consensus.cc:374] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.342960 12140 raft_consensus.cc:729] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.343055 11056 raft_consensus.cc:348] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.343096 12140 consensus_queue.cc:260] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.343147 11056 raft_consensus.cc:374] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.343178 11056 raft_consensus.cc:729] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.343259 12155 tablet_bootstrap.cc:654] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.343292 11056 consensus_queue.cc:260] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.343387 12140 ts_tablet_manager.cc:1398] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.343586 11056 ts_tablet_manager.cc:1398] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.345490 12155 tablet_bootstrap.cc:492] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:16.345564 12155 ts_tablet_manager.cc:1367] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:16.345985 12155 raft_consensus.cc:348] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.346086 12155 raft_consensus.cc:374] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.346114 12155 raft_consensus.cc:729] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.346227 12155 consensus_queue.cc:260] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.346478 12155 ts_tablet_manager.cc:1398] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.408085 12138 ts_tablet_manager.cc:902] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:16.408517 12138 tablet_copy_client.cc:308] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:16.408998 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet d4308bc87be24f28b6566ce76cae6e50 from peer c8d71787690245e681c85a087086e7ed ({username='slave'} at 127.0.99.133:52241)
I20240503 07:59:16.409090 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet d4308bc87be24f28b6566ce76cae6e50 from peer c8d71787690245e681c85a087086e7ed at {username='slave'} at 127.0.99.133:52241: session id = c8d71787690245e681c85a087086e7ed-d4308bc87be24f28b6566ce76cae6e50
I20240503 07:59:16.409818 6318 tablet_copy_source_session.cc:205] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:16.410627 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4308bc87be24f28b6566ce76cae6e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.412307 12138 tablet_copy_client.cc:783] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:16.412446 12138 tablet_copy_client.cc:647] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:16.413399 12138 tablet_copy_client.cc:523] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.414405 12138 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:16.438906 12138 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.439210 12138 tablet_bootstrap.cc:492] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Bootstrap complete.
I20240503 07:59:16.439360 12138 ts_tablet_manager.cc:1367] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 07:59:16.439754 12138 raft_consensus.cc:348] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:16.439882 12138 raft_consensus.cc:729] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: LEARNER
I20240503 07:59:16.439973 12138 consensus_queue.cc:260] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: NON_VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: true } }
I20240503 07:59:16.440254 12138 ts_tablet_manager.cc:1398] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:16.440668 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session c8d71787690245e681c85a087086e7ed-d4308bc87be24f28b6566ce76cae6e50 received from {username='slave'} at 127.0.99.133:52241
I20240503 07:59:16.440739 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session c8d71787690245e681c85a087086e7ed-d4308bc87be24f28b6566ce76cae6e50 on tablet d4308bc87be24f28b6566ce76cae6e50 with peer c8d71787690245e681c85a087086e7ed
I20240503 07:59:16.442914 8579 raft_consensus.cc:1203] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:16.462461 12426 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.466269 11589 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.468238 12035 consensus_queue.cc:1035] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.479255 11589 consensus_queue.cc:1035] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.480433 12240 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.481652 12035 raft_consensus.cc:482] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.481745 12035 raft_consensus.cc:504] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.482082 12035 leader_election.cc:288] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:16.482486 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a5be24d24419aa9388572b0bfd97d" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:16.482617 8711 raft_consensus.cc:2454] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:16.482766 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a5be24d24419aa9388572b0bfd97d" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.482906 8976 raft_consensus.cc:2454] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:16.483824 8517 leader_election.cc:302] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:16.484170 11589 raft_consensus.cc:2781] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.484222 11589 raft_consensus.cc:482] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.484256 11589 raft_consensus.cc:3037] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.485320 11589 raft_consensus.cc:504] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.485651 11589 leader_election.cc:288] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:16.486095 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a5be24d24419aa9388572b0bfd97d" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:16.486217 8711 raft_consensus.cc:3037] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.486248 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4a5be24d24419aa9388572b0bfd97d" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.486332 8977 raft_consensus.cc:3037] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.487169 8711 raft_consensus.cc:2454] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:16.487169 8977 raft_consensus.cc:2454] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:16.487545 8517 leader_election.cc:302] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:16.487725 12035 raft_consensus.cc:2781] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.487809 12035 raft_consensus.cc:686] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:16.487931 12035 consensus_queue.cc:237] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.491299 6149 catalog_manager.cc:5484] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.492128 9347 raft_consensus.cc:1050] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 3d08a06d53b14381bee6223c4e1a6027 to VOTER
I20240503 07:59:16.492545 9347 consensus_queue.cc:237] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:16.493510 11508 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.495033 6428 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.495373 6298 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.496124 8977 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.496245 12470 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.496834 10254 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.497572 12470 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.499640 6560 raft_consensus.cc:1203] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.501652 10254 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.502019 6428 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.504122 6298 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.504843 6151 catalog_manager.cc:5484] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.509492 8977 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.512585 6691 consensus_queue.cc:237] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } }
I20240503 07:59:16.514321 8977 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:16.514544 6298 raft_consensus.cc:1261] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:16.514627 9968 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:16.515703 8711 raft_consensus.cc:1203] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:16.516530 9968 consensus_queue.cc:1035] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:16.520165 9968 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.520372 6298 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.521764 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 558c0eb4a4334dbe91ac60fad4e820dc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.522728 8977 raft_consensus.cc:2932] T 558c0eb4a4334dbe91ac60fad4e820dc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } }
I20240503 07:59:16.523089 6155 catalog_manager.cc:5484] T 558c0eb4a4334dbe91ac60fad4e820dc P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.525405 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c0eb4a4334dbe91ac60fad4e820dc with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:16.525666 12438 tablet_replica.cc:330] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:16.525794 12438 raft_consensus.cc:2229] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.525946 12438 raft_consensus.cc:2258] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.526717 12438 ts_tablet_manager.cc:1875] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.529331 12438 ts_tablet_manager.cc:1888] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:16.529393 12438 log.cc:1196] T 558c0eb4a4334dbe91ac60fad4e820dc P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/558c0eb4a4334dbe91ac60fad4e820dc
I20240503 07:59:16.530449 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 558c0eb4a4334dbe91ac60fad4e820dc (table rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]) successfully deleted
I20240503 07:59:16.549988 12500 ts_tablet_manager.cc:902] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:16.550464 12500 tablet_copy_client.cc:308] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:16.551067 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet afeff5ce33bc4205a8599bbb66b24672 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:51161)
I20240503 07:59:16.551185 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet afeff5ce33bc4205a8599bbb66b24672 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:51161: session id = d1f36d57661b4429a4be8d10b58d8ff7-afeff5ce33bc4205a8599bbb66b24672
I20240503 07:59:16.552122 6581 tablet_copy_source_session.cc:205] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:16.553092 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeff5ce33bc4205a8599bbb66b24672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.556464 8579 raft_consensus.cc:1203] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:16.557981 12500 tablet_copy_client.cc:783] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:16.558164 12500 tablet_copy_client.cc:647] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:16.559504 12500 tablet_copy_client.cc:523] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.560681 12500 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:16.563747 12500 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.563944 12500 tablet_bootstrap.cc:492] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:16.564078 12500 ts_tablet_manager.cc:1367] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:16.564517 12500 raft_consensus.cc:348] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:16.564993 12500 raft_consensus.cc:729] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:16.565127 12500 consensus_queue.cc:260] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:16.565402 12500 ts_tablet_manager.cc:1398] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:16.566011 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-afeff5ce33bc4205a8599bbb66b24672 received from {username='slave'} at 127.0.99.130:51161
I20240503 07:59:16.566094 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-afeff5ce33bc4205a8599bbb66b24672 on tablet afeff5ce33bc4205a8599bbb66b24672 with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:16.597721 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:16.603559 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 6f260c4c4e7e4fcc8fbce11064bdc338 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.603633 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 6f260c4c4e7e4fcc8fbce11064bdc338 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.603863 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f260c4c4e7e4fcc8fbce11064bdc338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.603873 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f260c4c4e7e4fcc8fbce11064bdc338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.604449 8692 tablet_service.cc:1467] Processing CreateTablet for tablet fc7d26466129405bbe6db2721823bae1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.604449 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 6f260c4c4e7e4fcc8fbce11064bdc338 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.604748 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7d26466129405bbe6db2721823bae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.604894 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f260c4c4e7e4fcc8fbce11064bdc338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.606194 12140 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:16.606243 8690 tablet_service.cc:1467] Processing CreateTablet for tablet 2054e7790d7745c2a1b08e903c64d1b7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.606443 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054e7790d7745c2a1b08e903c64d1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.606822 12155 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:16.607050 12140 tablet_bootstrap.cc:654] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.607617 12155 tablet_bootstrap.cc:654] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.608124 12140 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:16.608198 12140 ts_tablet_manager.cc:1367] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:16.608633 12140 raft_consensus.cc:348] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.608737 12140 raft_consensus.cc:374] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.608799 12140 raft_consensus.cc:729] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.608831 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 2054e7790d7745c2a1b08e903c64d1b7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.608908 12140 consensus_queue.cc:260] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.609069 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054e7790d7745c2a1b08e903c64d1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.609181 12140 ts_tablet_manager.cc:1398] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.609189 8824 tablet_service.cc:1467] Processing CreateTablet for tablet fc7d26466129405bbe6db2721823bae1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.609419 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7d26466129405bbe6db2721823bae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.611198 12544 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:16.612151 12544 tablet_bootstrap.cc:654] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.613013 8955 tablet_service.cc:1467] Processing CreateTablet for tablet fc7d26466129405bbe6db2721823bae1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.613232 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7d26466129405bbe6db2721823bae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.614647 12545 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:16.615435 6149 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:16.615952 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2054e7790d7745c2a1b08e903c64d1b7 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=68cd74d1f907476d8cccfd32a1339623]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:16.616142 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054e7790d7745c2a1b08e903c64d1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.616333 12140 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:16.618945 12155 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:16.619033 12155 ts_tablet_manager.cc:1367] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:16.619096 12140 tablet_bootstrap.cc:654] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.619477 12155 raft_consensus.cc:348] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.619581 12155 raft_consensus.cc:374] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.619612 12155 raft_consensus.cc:729] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.619632 12541 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:16.619711 12155 consensus_queue.cc:260] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.619745 8711 raft_consensus.cc:1261] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.619974 12155 ts_tablet_manager.cc:1398] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.620114 12155 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:16.620112 12426 consensus_queue.cc:1035] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.620493 12541 tablet_bootstrap.cc:654] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.620890 12155 tablet_bootstrap.cc:654] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.621414 12545 tablet_bootstrap.cc:654] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.622200 8977 raft_consensus.cc:1261] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.622871 12035 consensus_queue.cc:1035] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:16.624537 12557 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.624878 12557 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:16.624912 6149 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]). Expected version 1 got 0
I20240503 07:59:16.625803 12541 tablet_bootstrap.cc:492] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:16.625919 12541 ts_tablet_manager.cc:1367] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:16.626708 12541 raft_consensus.cc:348] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.627545 6149 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]). Expected version 1 got 0
I20240503 07:59:16.628043 12556 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.628329 12556 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:16.629105 9400 raft_consensus.cc:1050] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:16.629410 12155 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:16.629473 12155 ts_tablet_manager.cc:1367] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20240503 07:59:16.629439 9400 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:16.629909 12541 raft_consensus.cc:374] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.629954 12541 raft_consensus.cc:729] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.630079 12541 consensus_queue.cc:260] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.630342 12541 ts_tablet_manager.cc:1398] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:16.630733 8842 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.629841 12155 raft_consensus.cc:348] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.630810 12155 raft_consensus.cc:374] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.631070 9189 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.631072 12155 raft_consensus.cc:729] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.631433 8579 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.631636 12155 consensus_queue.cc:260] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.631810 9189 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.632088 6298 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.632306 12155 ts_tablet_manager.cc:1398] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:16.632411 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e] alter complete (version 1)
I20240503 07:59:16.633015 12155 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:16.633602 9189 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.634066 12155 tablet_bootstrap.cc:654] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.634896 12544 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:16.635027 12544 ts_tablet_manager.cc:1367] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 07:59:16.635434 12544 raft_consensus.cc:348] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.635597 12544 raft_consensus.cc:374] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.636291 12544 raft_consensus.cc:729] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.636745 12544 consensus_queue.cc:260] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.637136 12544 ts_tablet_manager.cc:1398] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:16.639159 12155 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:16.639261 12155 ts_tablet_manager.cc:1367] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:16.639662 12155 raft_consensus.cc:348] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.639773 12155 raft_consensus.cc:374] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.639811 12155 raft_consensus.cc:729] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.639938 12155 consensus_queue.cc:260] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.640975 12155 ts_tablet_manager.cc:1398] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:16.641978 6151 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e]). Expected version 1 got 0
I20240503 07:59:16.642686 12140 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:16.642769 12140 ts_tablet_manager.cc:1367] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.002s
I20240503 07:59:16.643040 12565 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:16.643452 12140 raft_consensus.cc:348] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.643545 12140 raft_consensus.cc:374] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.643576 12140 raft_consensus.cc:729] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.643656 12545 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:16.643685 12140 consensus_queue.cc:260] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.643729 12545 ts_tablet_manager.cc:1367] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 07:59:16.643934 12140 ts_tablet_manager.cc:1398] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:16.644075 12140 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:16.644125 12545 raft_consensus.cc:348] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.644209 12545 raft_consensus.cc:374] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.644237 12545 raft_consensus.cc:729] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.644347 12545 consensus_queue.cc:260] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.644418 12565 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:16.644605 12545 ts_tablet_manager.cc:1398] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:16.644917 12140 tablet_bootstrap.cc:654] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:16.646564 11508 raft_consensus.cc:482] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.646641 11508 raft_consensus.cc:504] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.646962 11508 leader_election.cc:288] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.647436 9189 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.648360 8579 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.650194 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7d26466129405bbe6db2721823bae1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.650293 8976 raft_consensus.cc:2454] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:16.650604 8778 leader_election.cc:302] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:16.650756 11508 raft_consensus.cc:2781] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.650661 6151 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.650811 11508 raft_consensus.cc:482] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.650847 11508 raft_consensus.cc:3037] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.651072 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7d26466129405bbe6db2721823bae1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:16.651175 8711 raft_consensus.cc:2454] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:16.651717 11508 raft_consensus.cc:504] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.652055 11508 leader_election.cc:288] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.652554 6298 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.653146 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7d26466129405bbe6db2721823bae1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:16.653239 8711 raft_consensus.cc:3037] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.654067 8711 raft_consensus.cc:2454] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:16.654382 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7d26466129405bbe6db2721823bae1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.654477 8976 raft_consensus.cc:3037] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.655202 8976 raft_consensus.cc:2454] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:16.655401 8779 leader_election.cc:302] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:16.655841 11508 raft_consensus.cc:2781] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.655917 11508 raft_consensus.cc:686] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:16.656023 11508 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.656618 6691 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:16.658103 8843 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:16.658448 10254 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.658905 8711 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:16.659466 6149 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.659794 12140 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:16.659862 12140 ts_tablet_manager.cc:1367] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.001s
I20240503 07:59:16.660099 6428 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:16.660707 10254 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.660748 12140 raft_consensus.cc:348] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.660854 12140 raft_consensus.cc:374] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:16.660889 12140 raft_consensus.cc:729] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:16.660985 12140 consensus_queue.cc:260] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.661202 12140 ts_tablet_manager.cc:1398] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240503 07:59:16.661558 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 435f73fb9f384b10b353df19037bcac3. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.663061 8579 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.663466 6298 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.664022 9400 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.664501 10815 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.665458 8843 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.665633 9968 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.666209 6428 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.666779 10815 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.667850 8843 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.669265 8579 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.670127 6155 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.670883 6149 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.672716 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8cf3d6a3ace46769854ed61f340fb25 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.673585 12571 ts_tablet_manager.cc:902] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:16.674113 12571 tablet_copy_client.cc:308] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:16.674558 6298 raft_consensus.cc:2932] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:16.674990 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet a833fb4980c04673a335d020a3ad3dd3 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:38091)
I20240503 07:59:16.675078 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet a833fb4980c04673a335d020a3ad3dd3 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:38091: session id = ad910f92ef9d4db8a8962cecba289f8a-a833fb4980c04673a335d020a3ad3dd3
I20240503 07:59:16.675674 11370 raft_consensus.cc:1050] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:16.675926 8732 tablet_copy_source_session.cc:205] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:16.676043 11370 consensus_queue.cc:237] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:16.677178 6428 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.677361 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet c8cf3d6a3ace46769854ed61f340fb25 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:16.677392 6691 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.677632 12204 tablet_replica.cc:330] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:16.677772 12204 raft_consensus.cc:2229] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.677901 12204 raft_consensus.cc:2258] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.677953 12578 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.678901 12578 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.679049 12204 ts_tablet_manager.cc:1875] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.681057 12204 ts_tablet_manager.cc:1888] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:16.681123 12204 log.cc:1196] T c8cf3d6a3ace46769854ed61f340fb25 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/c8cf3d6a3ace46769854ed61f340fb25
I20240503 07:59:16.681123 12578 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.681427 6428 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.682880 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet c8cf3d6a3ace46769854ed61f340fb25 (table rebalancer_extra_table_0 [id=e85dfa9534124e9daa2fee791efcba8a]) successfully deleted
I20240503 07:59:16.683104 8977 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.683341 6691 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.683533 11370 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.683928 12571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a833fb4980c04673a335d020a3ad3dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:16.684911 8977 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.685154 6149 catalog_manager.cc:5484] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.688457 12571 tablet_copy_client.cc:783] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:16.688611 12571 tablet_copy_client.cc:647] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:16.689752 12571 tablet_copy_client.cc:523] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:16.690059 6298 consensus_queue.cc:237] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:16.690790 12571 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:16.690986 6428 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.691041 8711 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:16.691116 6691 raft_consensus.cc:1261] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.691545 12576 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.692256 8579 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.692260 11370 consensus_queue.cc:1035] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.692649 9189 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 07:59:16.693549 8648 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b61209b73f1f4872b5eccc823b1fa3df. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.693704 8843 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:16.694582 10079 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:16.695202 10079 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.695668 12579 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.695933 6430 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.696753 6691 raft_consensus.cc:2932] T e103e0e9cb16416dba7f8bbfc08ceac8 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.696748 8579 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.699234 6153 catalog_manager.cc:5484] T e103e0e9cb16416dba7f8bbfc08ceac8 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.699304 6157 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.700320 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e103e0e9cb16416dba7f8bbfc08ceac8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.701700 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet e103e0e9cb16416dba7f8bbfc08ceac8 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:16.701817 8843 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:16.704150 12571 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:16.704382 12571 tablet_bootstrap.cc:492] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:16.704617 12571 ts_tablet_manager.cc:1367] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240503 07:59:16.704795 12596 tablet_replica.cc:330] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:16.704958 12596 raft_consensus.cc:2229] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.705103 12571 raft_consensus.cc:348] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:16.705214 12571 raft_consensus.cc:729] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:16.705173 12596 raft_consensus.cc:2258] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.705312 12571 consensus_queue.cc:260] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:16.707123 12571 ts_tablet_manager.cc:1398] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:16.707592 12596 ts_tablet_manager.cc:1875] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.708138 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-a833fb4980c04673a335d020a3ad3dd3 received from {username='slave'} at 127.0.99.131:38091
I20240503 07:59:16.708213 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-a833fb4980c04673a335d020a3ad3dd3 on tablet a833fb4980c04673a335d020a3ad3dd3 with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:16.709662 12576 raft_consensus.cc:1050] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:16.710084 12576 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:16.710296 12596 ts_tablet_manager.cc:1888] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:16.710364 12596 log.cc:1196] T e103e0e9cb16416dba7f8bbfc08ceac8 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/e103e0e9cb16416dba7f8bbfc08ceac8
I20240503 07:59:16.710817 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet e103e0e9cb16416dba7f8bbfc08ceac8 (table rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]) successfully deleted
I20240503 07:59:16.711823 8579 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.712209 8711 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.712244 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.712462 12576 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.713106 12578 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.713272 12579 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.715562 12578 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.716040 8579 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.716636 9963 raft_consensus.cc:1050] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:16.717056 9963 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:16.717502 8711 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.718612 8843 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.718600 6151 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.718816 6691 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.719103 8579 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.719118 11648 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.719264 9963 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.719689 11648 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.721100 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.725342 6298 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:16.725978 11648 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.726119 6691 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.726716 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.727123 11370 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.727841 8711 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.727996 8579 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.726250 10254 raft_consensus.cc:482] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.728479 10254 raft_consensus.cc:504] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.729137 6157 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.729635 10254 leader_election.cc:288] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.729936 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.730062 8977 raft_consensus.cc:2454] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:16.730592 6627 leader_election.cc:302] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:16.730772 10254 raft_consensus.cc:2781] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.730819 10254 raft_consensus.cc:482] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.730865 10254 raft_consensus.cc:3037] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.730969 9189 raft_consensus.cc:482] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.731050 9189 raft_consensus.cc:504] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.731256 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:16.731365 8711 raft_consensus.cc:2454] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:16.731395 9189 leader_election.cc:288] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.731573 10254 raft_consensus.cc:504] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.731751 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.731881 8977 raft_consensus.cc:2454] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:16.731913 10254 leader_election.cc:288] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.732157 8647 leader_election.cc:302] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:16.732379 9189 raft_consensus.cc:2781] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.732432 9189 raft_consensus.cc:482] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.732471 9189 raft_consensus.cc:3037] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.732314 8843 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.732158 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:16.733008 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:16.733139 6691 raft_consensus.cc:2379] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9553fe3c4ccd4e91b7c176aad82fec09 in current term 1: Already voted for candidate a97073f89fd44702bccb74a9b76f858d in this term.
I20240503 07:59:16.733136 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.733215 8977 raft_consensus.cc:3037] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.733350 9189 raft_consensus.cc:504] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.733714 9189 leader_election.cc:288] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:16.733868 11370 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.734064 8977 raft_consensus.cc:2454] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:16.734464 6627 leader_election.cc:302] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:16.734773 8711 raft_consensus.cc:2379] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a97073f89fd44702bccb74a9b76f858d in current term 1: Already voted for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in this term.
I20240503 07:59:16.735862 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.736001 8977 raft_consensus.cc:2379] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9553fe3c4ccd4e91b7c176aad82fec09 in current term 1: Already voted for candidate a97073f89fd44702bccb74a9b76f858d in this term.
I20240503 07:59:16.736333 10254 raft_consensus.cc:2781] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.736414 10254 raft_consensus.cc:686] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:16.736318 11370 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.736495 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2054e7790d7745c2a1b08e903c64d1b7" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:16.736534 10254 consensus_queue.cc:237] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:16.736757 8647 leader_election.cc:302] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09; no voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:16.736922 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:16.737000 9189 raft_consensus.cc:2726] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 07:59:16.737978 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.738271 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551f25dcee034b7894498e3c748b0f0e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.738759 8712 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.741390 6150 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:16.743448 11589 raft_consensus.cc:1050] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:16.743584 12556 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.743872 12556 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:16.743837 11589 consensus_queue.cc:237] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:16.744151 12565 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.744457 12565 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:16.745288 6298 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.746279 12426 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.746467 6560 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.747005 12426 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.749079 8711 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.749965 12426 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.750105 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 551f25dcee034b7894498e3c748b0f0e with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:16.751561 12557 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:16.751843 12557 tablet.cc:1620] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:16.752238 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cd9526edb0a04b7fb3b6d8f123bcf07e] alter complete (version 2)
I20240503 07:59:16.754878 12426 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.756053 8711 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.755848 12626 tablet_replica.cc:330] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:16.756747 12626 raft_consensus.cc:2229] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.756901 12626 raft_consensus.cc:2258] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.757040 6298 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.757067 6559 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.758733 6560 consensus_queue.cc:237] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:16.759172 6149 catalog_manager.cc:5484] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.759490 12626 ts_tablet_manager.cc:1875] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.759781 8843 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:16.760159 9963 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.760143 6151 catalog_manager.cc:5484] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.761399 12626 ts_tablet_manager.cc:1888] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:16.761512 12626 log.cc:1196] T 551f25dcee034b7894498e3c748b0f0e P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/551f25dcee034b7894498e3c748b0f0e
I20240503 07:59:16.761730 8579 raft_consensus.cc:1261] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:16.762180 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 551f25dcee034b7894498e3c748b0f0e (table rebalancer_extra_table_2 [id=bc13ce7b7a704afbb59e119a72b305cd]) successfully deleted
I20240503 07:59:16.762326 12608 consensus_queue.cc:1035] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.764135 9963 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.765754 8579 consensus_queue.cc:237] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:16.765939 8843 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.767086 6560 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.767422 11589 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.768119 6298 raft_consensus.cc:1261] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.768508 11589 consensus_queue.cc:1035] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.768754 6151 catalog_manager.cc:5484] T c597bc1a19274f55b9d60558db35fcae P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.769150 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c597bc1a19274f55b9d60558db35fcae with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.770193 11589 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.771309 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet c597bc1a19274f55b9d60558db35fcae with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:16.771831 12634 tablet_replica.cc:330] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:16.771971 12634 raft_consensus.cc:2229] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.772169 12634 raft_consensus.cc:2258] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.772830 12634 ts_tablet_manager.cc:1875] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.772724 6560 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.774351 6298 raft_consensus.cc:2932] T c5b4c776eac24f739d85e09707cd751b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.775125 12634 ts_tablet_manager.cc:1888] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:16.775269 12634 log.cc:1196] T c597bc1a19274f55b9d60558db35fcae P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c597bc1a19274f55b9d60558db35fcae
I20240503 07:59:16.775724 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5b4c776eac24f739d85e09707cd751b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.775894 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet c597bc1a19274f55b9d60558db35fcae (table rebalance_test_table_13 [id=763a085c93d34fdbb045682646bf8b13]) successfully deleted
I20240503 07:59:16.776796 8579 raft_consensus.cc:2932] T c597bc1a19274f55b9d60558db35fcae P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:16.776968 6155 catalog_manager.cc:5484] T c5b4c776eac24f739d85e09707cd751b P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.779295 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b4c776eac24f739d85e09707cd751b with delete_type TABLET_DATA_TOMBSTONED (TS 9553fe3c4ccd4e91b7c176aad82fec09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:16.779673 12640 tablet_replica.cc:330] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:16.779796 12640 raft_consensus.cc:2229] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.779964 12640 raft_consensus.cc:2258] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.781901 12640 ts_tablet_manager.cc:1875] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.784351 12640 ts_tablet_manager.cc:1888] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:16.784421 12640 log.cc:1196] T c5b4c776eac24f739d85e09707cd751b P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/c5b4c776eac24f739d85e09707cd751b
I20240503 07:59:16.784902 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet c5b4c776eac24f739d85e09707cd751b (table rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]) successfully deleted
I20240503 07:59:16.787565 12434 raft_consensus.cc:1050] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:16.787971 12434 consensus_queue.cc:237] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:16.789225 6560 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.789498 6691 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.789873 12436 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.790050 8711 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:16.790060 12647 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.790436 12436 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:16.792563 12434 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.793118 8711 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.793598 6691 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.794690 6560 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.795851 6157 catalog_manager.cc:5484] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.798816 6430 consensus_queue.cc:237] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:16.800053 6560 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:16.800184 8711 raft_consensus.cc:1261] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:16.800472 12436 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:16.800647 9288 consensus_queue.cc:1035] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:16.802196 12434 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.802574 6560 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.803575 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 974c815ff8e94190a7bf28444cb3cd9f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.803851 8711 raft_consensus.cc:2932] T 974c815ff8e94190a7bf28444cb3cd9f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.805202 6151 catalog_manager.cc:5484] T 974c815ff8e94190a7bf28444cb3cd9f P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:16.807634 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 974c815ff8e94190a7bf28444cb3cd9f with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:16.807818 12634 tablet_replica.cc:330] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:16.807935 12634 raft_consensus.cc:2229] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.808084 12634 raft_consensus.cc:2258] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.809319 12634 ts_tablet_manager.cc:1875] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.810899 12634 ts_tablet_manager.cc:1888] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:16.810962 12634 log.cc:1196] T 974c815ff8e94190a7bf28444cb3cd9f P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/974c815ff8e94190a7bf28444cb3cd9f
I20240503 07:59:16.811604 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 974c815ff8e94190a7bf28444cb3cd9f (table rebalance_test_table_5 [id=936b932ca2ae410baea29e6202e71429]) successfully deleted
I20240503 07:59:16.830526 9189 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:16.830631 9189 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.831024 9189 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:16.831307 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:16.831423 6296 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:16.831609 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:16.831737 8645 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:16.831754 8977 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:16.831897 9189 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:16.831959 9189 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:16.831997 9189 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.833040 9189 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.833390 9189 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:16.833688 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:16.833807 8977 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.833875 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:16.833973 6296 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:16.834784 8977 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:16.834816 6296 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:16.835048 8647 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:16.835217 9189 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:16.835304 9189 raft_consensus.cc:686] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:16.835410 9189 consensus_queue.cc:237] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:16.837455 6155 catalog_manager.cc:5484] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.854686 6430 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:16.855919 6560 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:16.856077 8711 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:16.856132 6365 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Couldn't send request to peer 616fdda6ba714cb7be01a564de2b9db2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edf2c780d0d64db79dca88eac77d4478. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.856433 12436 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.856621 12647 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.857340 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:16.858451 12434 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:16.858851 8711 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:16.861730 6560 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } } }
I20240503 07:59:16.862154 6151 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.862826 12556 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.866165 12565 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.868516 6149 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=cd9526edb0a04b7fb3b6d8f123bcf07e] alter complete (version 3)
I20240503 07:59:16.869143 12557 tablet.cc:1722] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:16.875874 6559 consensus_queue.cc:237] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:16.877084 6430 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:16.877116 6296 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:16.877358 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afcd161598db41d7962caca61f63ef77. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.877439 12344 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.877585 11648 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:16.879269 12608 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.879662 6296 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.880879 6430 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.881928 6149 catalog_manager.cc:5484] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 4, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:16.897181 8579 raft_consensus.cc:1203] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:16.934543 10254 raft_consensus.cc:1050] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:16.934923 10254 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:16.936287 6559 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.936287 8579 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.936301 8843 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:16.936758 12470 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.936955 9968 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.937098 9347 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:16.938807 6430 raft_consensus.cc:1203] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:16.940649 9968 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.942173 8579 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.942149 8843 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.944242 6151 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.944842 6559 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.948413 6691 consensus_queue.cc:237] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:16.949784 8843 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:16.949877 8579 raft_consensus.cc:1261] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:16.950239 10254 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.950577 9347 consensus_queue.cc:1035] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:16.952152 10254 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.952795 8579 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.953676 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82de209205b342cfaac10506fdac9303 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:16.954090 8842 raft_consensus.cc:2932] T 82de209205b342cfaac10506fdac9303 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:16.954715 6156 catalog_manager.cc:5484] T 82de209205b342cfaac10506fdac9303 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.957176 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 82de209205b342cfaac10506fdac9303 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:16.958528 12767 tablet_replica.cc:330] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:16.958659 12767 raft_consensus.cc:2229] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.958833 12767 raft_consensus.cc:2258] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.959365 12767 ts_tablet_manager.cc:1875] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:16.961164 12767 ts_tablet_manager.cc:1888] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:16.961256 12767 log.cc:1196] T 82de209205b342cfaac10506fdac9303 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/82de209205b342cfaac10506fdac9303
I20240503 07:59:16.961691 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 82de209205b342cfaac10506fdac9303 (table rebalance_test_table_9 [id=e126a94a1cac419b895e54036cecb857]) successfully deleted
I20240503 07:59:16.970379 6149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.970855 6149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:16.973878 6691 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:16.974608 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 2e4a5be24d24419aa9388572b0bfd97d
I20240503 07:59:16.975127 8579 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:16.975347 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:16.975584 8692 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:16.975649 12596 tablet_replica.cc:330] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:16.975744 12640 tablet_replica.cc:330] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:16.975780 12596 raft_consensus.cc:2229] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.975872 12640 raft_consensus.cc:2229] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:16.975905 12596 raft_consensus.cc:2258] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.976003 12640 raft_consensus.cc:2258] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.976145 6560 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:16.976501 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9772b487ca864ad98e9d79d8435c34c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:16.976625 12470 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.976733 9189 raft_consensus.cc:1050] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:16.977032 12640 ts_tablet_manager.cc:1875] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.977030 9347 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:16.977128 12596 ts_tablet_manager.cc:1875] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.977167 9189 consensus_queue.cc:237] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:16.978411 6560 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.978714 8842 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.978787 10815 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.978978 12640 ts_tablet_manager.cc:1888] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.978978 12596 ts_tablet_manager.cc:1888] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.979040 12596 log.cc:1196] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2e4a5be24d24419aa9388572b0bfd97d
I20240503 07:59:16.979040 12640 log.cc:1196] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2e4a5be24d24419aa9388572b0bfd97d
I20240503 07:59:16.979269 12640 ts_tablet_manager.cc:1909] T 2e4a5be24d24419aa9388572b0bfd97d P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:16.979269 12596 ts_tablet_manager.cc:1909] T 2e4a5be24d24419aa9388572b0bfd97d P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:16.979389 6691 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:16.979921 10815 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.980230 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4a5be24d24419aa9388572b0bfd97d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:16 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:16.980386 12626 tablet_replica.cc:330] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:16.980499 12626 raft_consensus.cc:2229] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:16.980660 12626 raft_consensus.cc:2258] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:16.980849 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table_new_name [id=cd9526edb0a04b7fb3b6d8f123bcf07e]) successfully deleted
I20240503 07:59:16.980882 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table_new_name [id=cd9526edb0a04b7fb3b6d8f123bcf07e]) successfully deleted
I20240503 07:59:16.981788 12626 ts_tablet_manager.cc:1875] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:16.982033 10815 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:16.983481 12626 ts_tablet_manager.cc:1888] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:16.983541 12626 log.cc:1196] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e4a5be24d24419aa9388572b0bfd97d
I20240503 07:59:16.983736 12626 ts_tablet_manager.cc:1909] T 2e4a5be24d24419aa9388572b0bfd97d P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:16.983714 9400 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.984431 12470 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.985806 6690 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.986521 6560 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.986996 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 2e4a5be24d24419aa9388572b0bfd97d (table rebalancer_dynamic_table_new_name [id=cd9526edb0a04b7fb3b6d8f123bcf07e]) successfully deleted
I20240503 07:59:16.987114 6559 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.987944 8579 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:16.988726 6156 catalog_manager.cc:5484] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:16.989940 8842 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:16.990579 6155 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:17.035611 12647 raft_consensus.cc:1050] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:17.036048 12647 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:17.037392 6559 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.037395 8977 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.037864 12436 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.038183 12436 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.038436 8711 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.040267 9288 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.041699 9288 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.042280 6559 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.044476 6151 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:17.044601 8977 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.048696 8711 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.052603 6430 consensus_queue.cc:237] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:17.053704 8711 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:17.053723 8977 raft_consensus.cc:1261] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:17.054034 12436 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:17.054203 9288 consensus_queue.cc:1035] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:17.055708 12436 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.057381 8977 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.057610 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a10085c34be4d31aae9f56a2ec1ba23 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:17.059427 6155 catalog_manager.cc:5484] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.061195 8712 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09866b93f75545a19c9451556254da48"
dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60294
I20240503 07:59:17.061340 8712 raft_consensus.cc:595] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:17.061751 8711 raft_consensus.cc:2932] T 2a10085c34be4d31aae9f56a2ec1ba23 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:17.062925 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a10085c34be4d31aae9f56a2ec1ba23 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:17.063148 12767 tablet_replica.cc:330] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:17.063306 12767 raft_consensus.cc:2229] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.063519 12767 raft_consensus.cc:2258] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.064255 12767 ts_tablet_manager.cc:1875] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:17.066401 12767 ts_tablet_manager.cc:1888] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:17.066736 12767 log.cc:1196] T 2a10085c34be4d31aae9f56a2ec1ba23 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2a10085c34be4d31aae9f56a2ec1ba23
I20240503 07:59:17.067463 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 2a10085c34be4d31aae9f56a2ec1ba23 (table rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]) successfully deleted
I20240503 07:59:17.081331 6149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:17.086968 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 5178e51da98e4779bc7669e972e20606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]), partition=
I20240503 07:59:17.087220 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5178e51da98e4779bc7669e972e20606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.089062 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 5178e51da98e4779bc7669e972e20606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]), partition=
I20240503 07:59:17.089300 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5178e51da98e4779bc7669e972e20606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.089457 12854 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:17.090355 12854 tablet_bootstrap.cc:654] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.091074 12155 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.091658 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 5178e51da98e4779bc7669e972e20606 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]), partition=
I20240503 07:59:17.091921 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5178e51da98e4779bc7669e972e20606. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.091933 12155 tablet_bootstrap.cc:654] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.092953 12155 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:17.093039 12155 ts_tablet_manager.cc:1367] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:17.093479 12155 raft_consensus.cc:348] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.093570 12155 raft_consensus.cc:374] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.093596 12155 raft_consensus.cc:729] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.093701 12155 consensus_queue.cc:260] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.093955 12155 ts_tablet_manager.cc:1398] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.094161 12545 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:17.094465 12854 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:17.094545 12854 ts_tablet_manager.cc:1367] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:17.094966 12545 tablet_bootstrap.cc:654] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.094952 12854 raft_consensus.cc:348] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.095053 12854 raft_consensus.cc:374] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.095081 12854 raft_consensus.cc:729] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.095201 12854 consensus_queue.cc:260] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.095468 12854 ts_tablet_manager.cc:1398] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.096845 12545 tablet_bootstrap.cc:492] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:17.096948 12545 ts_tablet_manager.cc:1367] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.097574 12545 raft_consensus.cc:348] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.097688 12545 raft_consensus.cc:374] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.097723 12545 raft_consensus.cc:729] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.097843 12545 consensus_queue.cc:260] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.098163 12545 ts_tablet_manager.cc:1398] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.111845 6559 raft_consensus.cc:1203] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:17.128576 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.132308 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.135147 10254 raft_consensus.cc:1050] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:17.135545 10254 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.136884 8579 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.137275 9968 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.137388 6430 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.137713 9968 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.137866 8977 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.138293 9968 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.139621 9968 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.139904 6430 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.140441 8977 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.142993 6149 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.145849 8579 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.154621 8711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09866b93f75545a19c9451556254da48"
dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60312
I20240503 07:59:17.154727 8711 raft_consensus.cc:595] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:17.188110 12915 ts_tablet_manager.cc:902] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:17.188552 12915 tablet_copy_client.cc:308] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:17.188984 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet b61209b73f1f4872b5eccc823b1fa3df from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:41115)
I20240503 07:59:17.189076 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet b61209b73f1f4872b5eccc823b1fa3df from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:41115: session id = d1f36d57661b4429a4be8d10b58d8ff7-b61209b73f1f4872b5eccc823b1fa3df
I20240503 07:59:17.189759 8732 tablet_copy_source_session.cc:205] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.190482 12915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.192325 12915 tablet_copy_client.cc:783] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.192469 12915 tablet_copy_client.cc:647] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.193701 12915 tablet_copy_client.cc:523] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.194669 12915 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:17.197790 12915 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.197980 12915 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:17.198117 12915 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 07:59:17.198455 12915 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:17.198546 12915 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:17.198640 12915 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:17.198877 12915 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.199340 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-b61209b73f1f4872b5eccc823b1fa3df received from {username='slave'} at 127.0.99.130:41115
I20240503 07:59:17.199419 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-b61209b73f1f4872b5eccc823b1fa3df on tablet b61209b73f1f4872b5eccc823b1fa3df with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:17.245462 12921 ts_tablet_manager.cc:902] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:17.245934 12921 tablet_copy_client.cc:308] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:17.246436 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 435f73fb9f384b10b353df19037bcac3 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:38577)
I20240503 07:59:17.246536 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 435f73fb9f384b10b353df19037bcac3 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:38577: session id = 9553fe3c4ccd4e91b7c176aad82fec09-435f73fb9f384b10b353df19037bcac3
I20240503 07:59:17.247251 6712 tablet_copy_source_session.cc:205] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.247956 12921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435f73fb9f384b10b353df19037bcac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.249814 12921 tablet_copy_client.cc:783] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.249997 12921 tablet_copy_client.cc:647] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.250906 12921 tablet_copy_client.cc:523] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.251816 12921 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.253229 10254 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.256649 10254 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.273183 12921 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.273396 12921 tablet_bootstrap.cc:492] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:17.273558 12921 ts_tablet_manager.cc:1367] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20240503 07:59:17.273873 12921 raft_consensus.cc:348] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.273981 12921 raft_consensus.cc:729] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:17.274077 12921 consensus_queue.cc:260] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.274307 12921 ts_tablet_manager.cc:1398] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.274734 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-435f73fb9f384b10b353df19037bcac3 received from {username='slave'} at 127.0.99.134:38577
I20240503 07:59:17.274821 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-435f73fb9f384b10b353df19037bcac3 on tablet 435f73fb9f384b10b353df19037bcac3 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:17.324276 12931 ts_tablet_manager.cc:902] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:17.324811 12931 tablet_copy_client.cc:308] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:17.325219 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet edf2c780d0d64db79dca88eac77d4478 from peer 616fdda6ba714cb7be01a564de2b9db2 ({username='slave'} at 127.0.99.135:58189)
I20240503 07:59:17.325309 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet edf2c780d0d64db79dca88eac77d4478 from peer 616fdda6ba714cb7be01a564de2b9db2 at {username='slave'} at 127.0.99.135:58189: session id = 616fdda6ba714cb7be01a564de2b9db2-edf2c780d0d64db79dca88eac77d4478
I20240503 07:59:17.326288 6450 tablet_copy_source_session.cc:205] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.327230 12931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf2c780d0d64db79dca88eac77d4478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.329615 12931 tablet_copy_client.cc:783] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.329785 12931 tablet_copy_client.cc:647] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.330924 12931 tablet_copy_client.cc:523] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.331957 12931 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:17.337930 9400 raft_consensus.cc:482] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.338027 9400 raft_consensus.cc:504] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.338371 9400 leader_election.cc:288] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:17.338688 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5178e51da98e4779bc7669e972e20606" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:17.338804 8842 raft_consensus.cc:2454] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:17.338814 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5178e51da98e4779bc7669e972e20606" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:17.338956 8579 raft_consensus.cc:2454] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 0.
I20240503 07:59:17.339154 8647 leader_election.cc:302] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.339342 9400 raft_consensus.cc:2781] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.339383 9400 raft_consensus.cc:482] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.339426 9400 raft_consensus.cc:3037] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.340090 9400 raft_consensus.cc:504] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.340377 9400 leader_election.cc:288] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:17.340687 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5178e51da98e4779bc7669e972e20606" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:17.340687 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5178e51da98e4779bc7669e972e20606" candidate_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:17.340802 8579 raft_consensus.cc:3037] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.340813 8842 raft_consensus.cc:3037] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.341634 8842 raft_consensus.cc:2454] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:17.341655 8579 raft_consensus.cc:2454] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9553fe3c4ccd4e91b7c176aad82fec09 in term 1.
I20240503 07:59:17.341902 8647 leader_election.cc:302] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.342082 9400 raft_consensus.cc:2781] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.342160 9400 raft_consensus.cc:686] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Leader. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:17.342295 9400 consensus_queue.cc:237] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.343216 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:17.344265 6149 catalog_manager.cc:5484] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134). New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:17.345629 9400 consensus_queue.cc:1035] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.351992 8957 tablet_service.cc:1467] Processing CreateTablet for tablet f7442d12e0234db38d896c09a9e73162 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.352027 8824 tablet_service.cc:1467] Processing CreateTablet for tablet cfd4deb6a5104141bfa51fe4114bb708 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.352164 8560 tablet_service.cc:1467] Processing CreateTablet for tablet e12d0243dbb54bdfa1f584011a3db5b6 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.352257 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd4deb6a5104141bfa51fe4114bb708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.352285 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7442d12e0234db38d896c09a9e73162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.352391 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12d0243dbb54bdfa1f584011a3db5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.352914 9400 consensus_queue.cc:1035] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.354359 12545 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:17.355005 6670 tablet_service.cc:1467] Processing CreateTablet for tablet f7442d12e0234db38d896c09a9e73162 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.355065 6541 tablet_service.cc:1467] Processing CreateTablet for tablet f7442d12e0234db38d896c09a9e73162 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.355217 12545 tablet_bootstrap.cc:654] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.355253 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7442d12e0234db38d896c09a9e73162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.355284 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7442d12e0234db38d896c09a9e73162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.355919 6539 tablet_service.cc:1467] Processing CreateTablet for tablet cfd4deb6a5104141bfa51fe4114bb708 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.356089 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd4deb6a5104141bfa51fe4114bb708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.357362 12854 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:17.357470 8955 tablet_service.cc:1467] Processing CreateTablet for tablet e12d0243dbb54bdfa1f584011a3db5b6 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.357535 12945 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:17.357668 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12d0243dbb54bdfa1f584011a3db5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.358150 12854 tablet_bootstrap.cc:654] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.358597 12944 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:17.358927 12545 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:17.359002 12545 ts_tablet_manager.cc:1367] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:17.359392 12545 raft_consensus.cc:348] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.359493 12545 raft_consensus.cc:374] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.359524 12545 raft_consensus.cc:729] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.359624 12545 consensus_queue.cc:260] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.359773 12945 tablet_bootstrap.cc:654] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.359876 12545 ts_tablet_manager.cc:1398] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.360361 12944 tablet_bootstrap.cc:654] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.360980 8692 tablet_service.cc:1467] Processing CreateTablet for tablet cfd4deb6a5104141bfa51fe4114bb708 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.361193 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd4deb6a5104141bfa51fe4114bb708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.362980 8690 tablet_service.cc:1467] Processing CreateTablet for tablet e12d0243dbb54bdfa1f584011a3db5b6 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:17.363137 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12d0243dbb54bdfa1f584011a3db5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.364305 12931 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.364521 12931 tablet_bootstrap.cc:492] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap complete.
I20240503 07:59:17.364662 12931 ts_tablet_manager.cc:1367] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.000s
I20240503 07:59:17.364939 12155 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.365043 12931 raft_consensus.cc:348] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:17.365157 12931 raft_consensus.cc:729] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: LEARNER
I20240503 07:59:17.365242 12931 consensus_queue.cc:260] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: NON_VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: true } }
I20240503 07:59:17.365516 12931 ts_tablet_manager.cc:1398] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:17.365732 12155 tablet_bootstrap.cc:654] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.366089 12854 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:17.366161 12854 ts_tablet_manager.cc:1367] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:17.366595 12854 raft_consensus.cc:348] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.366683 12854 raft_consensus.cc:374] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.366662 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session 616fdda6ba714cb7be01a564de2b9db2-edf2c780d0d64db79dca88eac77d4478 received from {username='slave'} at 127.0.99.135:58189
I20240503 07:59:17.366714 12854 raft_consensus.cc:729] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.366734 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session 616fdda6ba714cb7be01a564de2b9db2-edf2c780d0d64db79dca88eac77d4478 on tablet edf2c780d0d64db79dca88eac77d4478 with peer 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:17.366819 12854 consensus_queue.cc:260] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.366874 12155 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:17.366936 12155 ts_tablet_manager.cc:1367] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:17.367085 12854 ts_tablet_manager.cc:1398] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.367314 12155 raft_consensus.cc:348] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.367425 12155 raft_consensus.cc:374] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.367462 12155 raft_consensus.cc:729] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.367566 12155 consensus_queue.cc:260] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.367820 12155 ts_tablet_manager.cc:1398] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:17.367970 12155 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.368392 12942 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:17.368708 12155 tablet_bootstrap.cc:654] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.369256 12942 tablet_bootstrap.cc:654] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.370277 12945 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:17.370375 12945 ts_tablet_manager.cc:1367] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:17.370815 12942 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:17.370961 12942 ts_tablet_manager.cc:1367] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:17.371043 12945 raft_consensus.cc:348] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.371181 12945 raft_consensus.cc:374] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.371244 12945 raft_consensus.cc:729] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.371390 12945 consensus_queue.cc:260] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.371680 12945 ts_tablet_manager.cc:1398] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.372606 12942 raft_consensus.cc:348] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.372757 12942 raft_consensus.cc:374] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.372856 12942 raft_consensus.cc:729] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.373279 12942 consensus_queue.cc:260] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.374184 12035 raft_consensus.cc:482] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.374243 12942 ts_tablet_manager.cc:1398] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:17.374255 12035 raft_consensus.cc:504] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.374573 12035 leader_election.cc:288] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:17.374769 12942 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:17.374965 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:17.375321 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
W20240503 07:59:17.375643 8517 leader_election.cc:341] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Illegal state: must be running to vote when last-logged opid is not known
W20240503 07:59:17.376147 8516 leader_election.cc:341] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:17.376227 8516 leader_election.cc:302] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8d71787690245e681c85a087086e7ed; no voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:17.376374 12035 raft_consensus.cc:2726] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:17.378162 12942 tablet_bootstrap.cc:654] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.379540 12944 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:17.379627 12944 ts_tablet_manager.cc:1367] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:17.380071 12944 raft_consensus.cc:348] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.380167 12944 raft_consensus.cc:374] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.380196 12944 raft_consensus.cc:729] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.380316 12944 consensus_queue.cc:260] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.380584 12944 ts_tablet_manager.cc:1398] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:17.380724 12944 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:17.380856 12942 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:17.380925 12942 ts_tablet_manager.cc:1367] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:17.381132 12155 tablet_bootstrap.cc:492] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:17.381187 12155 ts_tablet_manager.cc:1367] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:17.381292 12942 raft_consensus.cc:348] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.381376 12942 raft_consensus.cc:374] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.381408 12942 raft_consensus.cc:729] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.381520 12942 consensus_queue.cc:260] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.381580 12155 raft_consensus.cc:348] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.381664 12155 raft_consensus.cc:374] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.381693 12155 raft_consensus.cc:729] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.381819 12942 ts_tablet_manager.cc:1398] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.381806 12155 consensus_queue.cc:260] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.382081 12155 ts_tablet_manager.cc:1398] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.383738 12944 tablet_bootstrap.cc:654] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.385529 12944 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:17.385659 12944 ts_tablet_manager.cc:1367] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:17.386066 12944 raft_consensus.cc:348] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.386164 12944 raft_consensus.cc:374] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.386194 12944 raft_consensus.cc:729] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.386313 12944 consensus_queue.cc:260] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.386653 12944 ts_tablet_manager.cc:1398] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.400820 11524 raft_consensus.cc:482] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.400909 11524 raft_consensus.cc:504] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.401270 11524 leader_election.cc:288] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:17.401495 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:17.401605 8711 raft_consensus.cc:2454] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:17.401661 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:17.401769 8579 raft_consensus.cc:2454] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:17.401865 8913 leader_election.cc:302] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.402309 11524 raft_consensus.cc:2781] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.402382 11524 raft_consensus.cc:482] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.402423 11524 raft_consensus.cc:3037] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.403252 11524 raft_consensus.cc:504] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.403543 11524 leader_election.cc:288] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:17.403791 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:17.403908 8711 raft_consensus.cc:3037] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.404014 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12d0243dbb54bdfa1f584011a3db5b6" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:17.404116 8579 raft_consensus.cc:3037] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.404831 8711 raft_consensus.cc:2454] T e12d0243dbb54bdfa1f584011a3db5b6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:17.405019 8579 raft_consensus.cc:2454] T e12d0243dbb54bdfa1f584011a3db5b6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:17.405138 8913 leader_election.cc:302] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.405287 11524 raft_consensus.cc:2781] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.405391 11524 raft_consensus.cc:686] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:17.405520 11524 consensus_queue.cc:237] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:17.407193 6156 catalog_manager.cc:5484] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:17.414084 9189 raft_consensus.cc:979] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: : Instructing follower ad910f92ef9d4db8a8962cecba289f8a to start an election
I20240503 07:59:17.414173 9189 raft_consensus.cc:1067] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Signalling peer ad910f92ef9d4db8a8962cecba289f8a to start an election
I20240503 07:59:17.414670 6560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09866b93f75545a19c9451556254da48"
dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
from {username='slave'} at 127.0.99.134:43151
I20240503 07:59:17.414786 6560 raft_consensus.cc:482] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:17.414850 6560 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.415578 6560 raft_consensus.cc:504] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:17.415923 6560 leader_election.cc:288] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:17.416143 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:17.416196 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:17.416173 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09866b93f75545a19c9451556254da48" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:17.416237 8711 raft_consensus.cc:3032] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:17.416270 8711 raft_consensus.cc:729] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Running, Role: LEADER
I20240503 07:59:17.416294 8842 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.416306 6690 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.416410 8711 consensus_queue.cc:260] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:17.416569 8711 raft_consensus.cc:3037] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.417137 6690 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:17.417271 8842 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:17.417440 8711 raft_consensus.cc:2454] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 2.
I20240503 07:59:17.417613 6496 leader_election.cc:302] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:17.417774 12608 raft_consensus.cc:2781] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:17.417851 12608 raft_consensus.cc:686] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 2 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:17.417966 12608 consensus_queue.cc:237] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:17.419592 6156 catalog_manager.cc:5484] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 2 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.425503 6690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35"
dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34194
I20240503 07:59:17.425642 6690 raft_consensus.cc:595] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Received request to transfer leadership
I20240503 07:59:17.432186 12921 ts_tablet_manager.cc:902] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:17.432600 12921 tablet_copy_client.cc:308] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:17.433187 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet afcd161598db41d7962caca61f63ef77 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:43151)
I20240503 07:59:17.433267 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet afcd161598db41d7962caca61f63ef77 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:43151: session id = 9553fe3c4ccd4e91b7c176aad82fec09-afcd161598db41d7962caca61f63ef77
I20240503 07:59:17.434150 6581 tablet_copy_source_session.cc:205] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.434967 12921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afcd161598db41d7962caca61f63ef77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.436923 12921 tablet_copy_client.cc:783] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.437055 12921 tablet_copy_client.cc:647] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.437899 12921 tablet_copy_client.cc:523] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.438941 12921 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.459879 12921 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.460065 12921 tablet_bootstrap.cc:492] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:17.460177 12921 ts_tablet_manager.cc:1367] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240503 07:59:17.460472 12921 raft_consensus.cc:348] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.460572 12921 raft_consensus.cc:729] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:17.460650 12921 consensus_queue.cc:260] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.460893 12921 ts_tablet_manager.cc:1398] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 07:59:17.461297 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-afcd161598db41d7962caca61f63ef77 received from {username='slave'} at 127.0.99.134:43151
I20240503 07:59:17.461360 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-afcd161598db41d7962caca61f63ef77 on tablet afcd161598db41d7962caca61f63ef77 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:17.486114 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:17.489539 8579 raft_consensus.cc:1261] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.489591 8842 raft_consensus.cc:1261] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:17.489699 12344 raft_consensus.cc:1050] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:17.489934 9189 consensus_queue.cc:1035] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.490115 12344 consensus_queue.cc:237] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:17.490334 9189 consensus_queue.cc:1035] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.491554 8843 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.491899 6296 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.493674 6157 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]). Expected version 1 got 0
I20240503 07:59:17.492949 13018 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.494040 6430 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.494112 13021 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.494396 13021 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:17.494603 12608 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.495271 13022 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.495571 13022 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:17.497005 13025 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:17.497061 6157 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]). Expected version 1 got 0
I20240503 07:59:17.497275 13025 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:17.497262 13018 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.499619 12608 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.500026 6156 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b]). Expected version 1 got 0
I20240503 07:59:17.500064 6430 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.500195 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b] alter complete (version 1)
I20240503 07:59:17.502166 6298 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.504882 6157 catalog_manager.cc:5484] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.507362 6560 consensus_queue.cc:237] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:17.508432 6430 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.508788 13017 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.509375 6298 raft_consensus.cc:1261] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.509721 13017 consensus_queue.cc:1035] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.510214 13018 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.510496 6430 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.511595 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afeff5ce33bc4205a8599bbb66b24672 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:17.512692 6149 catalog_manager.cc:5484] T afeff5ce33bc4205a8599bbb66b24672 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.513123 6298 raft_consensus.cc:2932] T afeff5ce33bc4205a8599bbb66b24672 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:17.514786 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff5ce33bc4205a8599bbb66b24672 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:17.515095 13027 tablet_replica.cc:330] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:17.515283 13027 raft_consensus.cc:2229] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.515399 13027 pending_rounds.cc:62] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Trying to abort 1 pending ops.
I20240503 07:59:17.515462 13027 pending_rounds.cc:68] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023506053078384640 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "afeff5ce33bc4205a8599bbb66b24672" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } } }
I20240503 07:59:17.515631 13027 raft_consensus.cc:2894] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 07:59:17.515702 13027 raft_consensus.cc:2866] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 07:59:17.515760 13027 raft_consensus.cc:2258] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.516268 13027 ts_tablet_manager.cc:1875] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:17.518201 13027 ts_tablet_manager.cc:1888] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:17.518267 13027 log.cc:1196] T afeff5ce33bc4205a8599bbb66b24672 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/afeff5ce33bc4205a8599bbb66b24672
I20240503 07:59:17.518730 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet afeff5ce33bc4205a8599bbb66b24672 (table rebalancer_extra_table_6 [id=bfcfc693f3d8455f9118e86b6a99ee80]) successfully deleted
I20240503 07:59:17.520120 12921 ts_tablet_manager.cc:902] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:17.520148 12579 raft_consensus.cc:1050] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER c8d71787690245e681c85a087086e7ed to VOTER
I20240503 07:59:17.520501 12579 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.520615 12921 tablet_copy_client.cc:308] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:17.521044 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 9772b487ca864ad98e9d79d8435c34c6 from peer 9553fe3c4ccd4e91b7c176aad82fec09 ({username='slave'} at 127.0.99.134:38577)
I20240503 07:59:17.521142 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 9772b487ca864ad98e9d79d8435c34c6 from peer 9553fe3c4ccd4e91b7c176aad82fec09 at {username='slave'} at 127.0.99.134:38577: session id = 9553fe3c4ccd4e91b7c176aad82fec09-9772b487ca864ad98e9d79d8435c34c6
I20240503 07:59:17.521979 6712 tablet_copy_source_session.cc:205] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:17.522024 8842 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.522274 6690 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.522581 8579 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:17.522596 12578 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.523135 12578 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.523486 12578 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:17.523720 12921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9772b487ca864ad98e9d79d8435c34c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.525434 12921 tablet_copy_client.cc:783] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:17.525576 12921 tablet_copy_client.cc:647] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:17.527220 12921 tablet_copy_client.cc:523] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:17.527145 13034 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.528236 12921 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:17.528268 6690 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.528466 8579 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.530666 6149 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.530781 8842 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8d71787690245e681c85a087086e7ed (127.0.99.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.534816 6298 consensus_queue.cc:237] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.536070 8842 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:17.536162 8579 raft_consensus.cc:1261] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:17.536418 13033 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:17.536581 13034 consensus_queue.cc:1035] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:17.537979 13038 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.538367 8842 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.538607 8579 raft_consensus.cc:2932] T d4308bc87be24f28b6566ce76cae6e50 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.540426 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4308bc87be24f28b6566ce76cae6e50 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:17.540581 6151 catalog_manager.cc:5484] T d4308bc87be24f28b6566ce76cae6e50 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 8 to 9, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:17.543203 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet d4308bc87be24f28b6566ce76cae6e50 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:17.543498 13041 tablet_replica.cc:330] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:17.543618 13041 raft_consensus.cc:2229] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.543815 13041 raft_consensus.cc:2258] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.544425 13041 ts_tablet_manager.cc:1875] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:17.546500 13041 ts_tablet_manager.cc:1888] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:17.546561 13041 log.cc:1196] T d4308bc87be24f28b6566ce76cae6e50 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4308bc87be24f28b6566ce76cae6e50
I20240503 07:59:17.547005 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet d4308bc87be24f28b6566ce76cae6e50 (table rebalance_test_table_11 [id=1b729dc47d2647d7b16c8da03ad3310c]) successfully deleted
I20240503 07:59:17.550818 12921 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:17.551008 12921 tablet_bootstrap.cc:492] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap complete.
I20240503 07:59:17.551120 12921 ts_tablet_manager.cc:1367] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20240503 07:59:17.551429 12921 raft_consensus.cc:348] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.551517 12921 raft_consensus.cc:729] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: LEARNER
I20240503 07:59:17.551587 12921 consensus_queue.cc:260] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:17.551908 12921 ts_tablet_manager.cc:1398] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 07:59:17.552484 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-9772b487ca864ad98e9d79d8435c34c6 received from {username='slave'} at 127.0.99.134:38577
I20240503 07:59:17.552542 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session 9553fe3c4ccd4e91b7c176aad82fec09-9772b487ca864ad98e9d79d8435c34c6 on tablet 9772b487ca864ad98e9d79d8435c34c6 with peer 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:17.572095 9347 raft_consensus.cc:979] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: : Instructing follower d1f36d57661b4429a4be8d10b58d8ff7 to start an election
I20240503 07:59:17.572191 9347 raft_consensus.cc:1067] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Signalling peer d1f36d57661b4429a4be8d10b58d8ff7 to start an election
I20240503 07:59:17.572696 6430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35"
dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
from {username='slave'} at 127.0.99.132:38973
I20240503 07:59:17.572852 6430 raft_consensus.cc:482] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:17.572851 12608 raft_consensus.cc:482] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.572909 6430 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.572930 12608 raft_consensus.cc:504] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.573266 12608 leader_election.cc:288] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:17.573562 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7442d12e0234db38d896c09a9e73162" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:17.573688 6690 raft_consensus.cc:2454] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:17.573912 6496 leader_election.cc:302] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:17.573890 6430 raft_consensus.cc:504] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.574067 12608 raft_consensus.cc:2781] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.574117 12608 raft_consensus.cc:482] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.574157 12608 raft_consensus.cc:3037] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.574342 6430 leader_election.cc:288] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:17.574700 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:17.574810 8575 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.574972 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:17.575083 8976 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.575083 12608 raft_consensus.cc:504] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.575384 12608 leader_election.cc:288] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:17.575726 8575 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 2.
I20240503 07:59:17.575824 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7442d12e0234db38d896c09a9e73162" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:17.575881 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7442d12e0234db38d896c09a9e73162" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:17.575971 6690 raft_consensus.cc:3037] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.575963 8975 raft_consensus.cc:3037] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.576192 8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7442d12e0234db38d896c09a9e73162" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:17.575845 8976 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 2.
I20240503 07:59:17.576679 6365 leader_election.cc:302] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:17.576697 6690 raft_consensus.cc:2454] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:17.576859 12434 raft_consensus.cc:2781] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:17.576907 8975 raft_consensus.cc:2454] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:17.576941 12434 raft_consensus.cc:686] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:17.577073 12434 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.577175 6496 leader_election.cc:302] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:17.577328 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8861b52c306e43d8b0615202cf7d8e35" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:17.577420 6690 raft_consensus.cc:3032] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:17.577456 6690 raft_consensus.cc:729] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:17.577446 12608 raft_consensus.cc:2781] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.577522 12608 raft_consensus.cc:686] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:17.577589 6690 consensus_queue.cc:260] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:17.577626 12608 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:17.577764 6690 raft_consensus.cc:3037] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:17.578857 6690 raft_consensus.cc:2454] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 2.
I20240503 07:59:17.579468 6156 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 1 to 2, leader changed from a97073f89fd44702bccb74a9b76f858d (127.0.99.132) to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:17.579545 6157 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.603869 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:17.608860 13021 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.609153 13021 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:17.609233 13025 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.609503 13025 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:17.609700 13022 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:17.609954 13022 tablet.cc:1620] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:17.614123 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ccdfd6b220004a9b96f0252b4b77a57b] alter complete (version 2)
I20240503 07:59:17.675108 11508 raft_consensus.cc:482] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.675220 11508 raft_consensus.cc:504] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.675626 11508 leader_election.cc:288] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:17.675822 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd4deb6a5104141bfa51fe4114bb708" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:17.675961 8711 raft_consensus.cc:2454] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:17.676158 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd4deb6a5104141bfa51fe4114bb708" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:17.676281 6560 raft_consensus.cc:2454] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:17.676330 8779 leader_election.cc:302] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.676578 11508 raft_consensus.cc:2781] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.676635 11508 raft_consensus.cc:482] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.676671 11508 raft_consensus.cc:3037] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.677659 11508 raft_consensus.cc:504] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.677981 11508 leader_election.cc:288] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:17.678215 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd4deb6a5104141bfa51fe4114bb708" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:17.678339 8711 raft_consensus.cc:3037] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.678508 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd4deb6a5104141bfa51fe4114bb708" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:17.678620 6560 raft_consensus.cc:3037] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.679253 8711 raft_consensus.cc:2454] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:17.679559 6560 raft_consensus.cc:2454] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:17.679589 8779 leader_election.cc:302] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:17.679816 11508 raft_consensus.cc:2781] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.679906 11508 raft_consensus.cc:686] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:17.680025 11508 consensus_queue.cc:237] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.680995 10079 raft_consensus.cc:1050] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:17.681341 10079 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:17.681943 6156 catalog_manager.cc:5484] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.682739 6560 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.682796 8842 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.683183 13080 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.683255 8975 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:17.683842 13080 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.684130 9189 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:17.686410 9400 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.686869 6560 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.687698 8975 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.688283 8842 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.689283 6156 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.693248 8711 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:17.694283 6560 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.694718 9400 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.694918 8975 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:17.695394 9400 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.697132 9189 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.698216 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a833fb4980c04673a335d020a3ad3dd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:17.698542 6560 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.699437 6430 raft_consensus.cc:1203] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:17.699518 6149 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:17.700951 8975 raft_consensus.cc:2932] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:17.703635 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet a833fb4980c04673a335d020a3ad3dd3 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:17.703819 13027 tablet_replica.cc:330] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:17.703938 13027 raft_consensus.cc:2229] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.704051 13027 raft_consensus.cc:2258] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.704581 13027 ts_tablet_manager.cc:1875] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:17.706638 13027 ts_tablet_manager.cc:1888] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:17.706693 13027 log.cc:1196] T a833fb4980c04673a335d020a3ad3dd3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a833fb4980c04673a335d020a3ad3dd3
I20240503 07:59:17.707156 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet a833fb4980c04673a335d020a3ad3dd3 (table rebalancer_extra_table_10 [id=aa6153f2da19423fbf58eba4c320afa3]) successfully deleted
I20240503 07:59:17.719678 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:17.724853 13021 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.725546 13022 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.726176 13025 tablet.cc:1722] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:17.729872 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=ccdfd6b220004a9b96f0252b4b77a57b] alter complete (version 3)
I20240503 07:59:17.731485 8842 raft_consensus.cc:1203] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:17.762934 8711 raft_consensus.cc:1203] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:17.833720 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:17.833848 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:17.835822 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 5178e51da98e4779bc7669e972e20606
I20240503 07:59:17.836426 8690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5178e51da98e4779bc7669e972e20606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:17.836411 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 5178e51da98e4779bc7669e972e20606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:17.836563 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5178e51da98e4779bc7669e972e20606 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:17 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:17.836614 13027 tablet_replica.cc:330] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:17.836707 13176 tablet_replica.cc:330] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:17.836824 13177 tablet_replica.cc:330] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:17.836858 13176 raft_consensus.cc:2229] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:17.836755 13027 raft_consensus.cc:2229] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.836987 13177 raft_consensus.cc:2229] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:17.836990 13027 raft_consensus.cc:2258] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.837082 13176 raft_consensus.cc:2258] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.837105 13177 raft_consensus.cc:2258] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:17.837592 13176 ts_tablet_manager.cc:1875] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.837694 13027 ts_tablet_manager.cc:1875] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.839469 13177 ts_tablet_manager.cc:1875] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:17.839563 13176 ts_tablet_manager.cc:1888] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.839563 13027 ts_tablet_manager.cc:1888] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.839641 13027 log.cc:1196] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/5178e51da98e4779bc7669e972e20606
I20240503 07:59:17.839641 13176 log.cc:1196] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5178e51da98e4779bc7669e972e20606
I20240503 07:59:17.839885 13027 ts_tablet_manager.cc:1909] T 5178e51da98e4779bc7669e972e20606 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:17.839927 13176 ts_tablet_manager.cc:1909] T 5178e51da98e4779bc7669e972e20606 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:17.840466 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table_new_name [id=ccdfd6b220004a9b96f0252b4b77a57b]) successfully deleted
I20240503 07:59:17.840512 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table_new_name [id=ccdfd6b220004a9b96f0252b4b77a57b]) successfully deleted
I20240503 07:59:17.841178 13177 ts_tablet_manager.cc:1888] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:17.841235 13177 log.cc:1196] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5178e51da98e4779bc7669e972e20606
I20240503 07:59:17.841400 13177 ts_tablet_manager.cc:1909] T 5178e51da98e4779bc7669e972e20606 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:17.842067 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 5178e51da98e4779bc7669e972e20606 (table rebalancer_dynamic_table_new_name [id=ccdfd6b220004a9b96f0252b4b77a57b]) successfully deleted
I20240503 07:59:17.868347 8711 raft_consensus.cc:1203] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 07:59:17.870528 11524 consensus_queue.cc:1035] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.873484 11524 consensus_queue.cc:1035] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:17.880534 8711 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:17.880903 13017 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.937450 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:17.942373 8955 tablet_service.cc:1467] Processing CreateTablet for tablet ae708f8882794827a03d9fe4a7e6777c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]), partition=
I20240503 07:59:17.942539 6277 tablet_service.cc:1467] Processing CreateTablet for tablet ae708f8882794827a03d9fe4a7e6777c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]), partition=
I20240503 07:59:17.942541 8824 tablet_service.cc:1467] Processing CreateTablet for tablet ae708f8882794827a03d9fe4a7e6777c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]), partition=
I20240503 07:59:17.942643 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae708f8882794827a03d9fe4a7e6777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.942760 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae708f8882794827a03d9fe4a7e6777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.942760 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae708f8882794827a03d9fe4a7e6777c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:17.944466 13182 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:17.945272 13184 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:17.946138 13184 tablet_bootstrap.cc:654] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.946336 13182 tablet_bootstrap.cc:654] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.946730 8711 raft_consensus.cc:1203] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:17.947031 13183 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:17.947676 13184 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:17.947777 13184 ts_tablet_manager.cc:1367] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.948246 13183 tablet_bootstrap.cc:654] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:17.948208 13184 raft_consensus.cc:348] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.948343 13184 raft_consensus.cc:374] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.948398 13184 raft_consensus.cc:729] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.948529 13184 consensus_queue.cc:260] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.948920 13184 ts_tablet_manager.cc:1398] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.949642 13182 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:17.949769 13182 ts_tablet_manager.cc:1367] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:17.950204 13182 raft_consensus.cc:348] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.950335 13183 tablet_bootstrap.cc:492] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:17.950356 13182 raft_consensus.cc:374] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.950414 13182 raft_consensus.cc:729] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.950464 13183 ts_tablet_manager.cc:1367] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:17.950558 13182 consensus_queue.cc:260] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.950884 13182 ts_tablet_manager.cc:1398] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.950923 13183 raft_consensus.cc:348] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.951023 13183 raft_consensus.cc:374] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:17.951081 13183 raft_consensus.cc:729] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:17.951293 13183 consensus_queue.cc:260] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.951608 13183 ts_tablet_manager.cc:1398] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:17.975611 6690 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:17.975972 12608 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.978806 8842 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:17.979329 12608 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:17.989173 11524 raft_consensus.cc:482] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:17.989248 11524 raft_consensus.cc:504] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.989590 11524 leader_election.cc:288] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:17.989846 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae708f8882794827a03d9fe4a7e6777c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:17.989966 6298 raft_consensus.cc:2454] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:17.989970 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae708f8882794827a03d9fe4a7e6777c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:17.990083 8842 raft_consensus.cc:2454] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:17.990211 8910 leader_election.cc:302] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:17.990423 11524 raft_consensus.cc:2781] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:17.990482 11524 raft_consensus.cc:482] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:17.990523 11524 raft_consensus.cc:3037] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.991252 11524 raft_consensus.cc:504] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.991547 11524 leader_election.cc:288] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:17.991911 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae708f8882794827a03d9fe4a7e6777c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:17.992049 6298 raft_consensus.cc:3037] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.992205 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae708f8882794827a03d9fe4a7e6777c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:17.992306 8842 raft_consensus.cc:3037] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:17.993016 6298 raft_consensus.cc:2454] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:17.993182 8842 raft_consensus.cc:2454] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:17.993271 8910 leader_election.cc:302] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:17.993502 11524 raft_consensus.cc:2781] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:17.993590 11524 raft_consensus.cc:686] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:17.993722 11524 consensus_queue.cc:237] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:17.995453 6151 catalog_manager.cc:5484] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.054324 8575 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:18.054673 13114 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.068713 12608 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.072160 13017 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.100780 11508 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.103343 11508 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.107879 6157 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:18.111449 8842 raft_consensus.cc:1261] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.111510 6298 raft_consensus.cc:1261] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.111824 13180 consensus_queue.cc:1035] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.111955 11524 consensus_queue.cc:1035] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.114208 13233 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.114552 13233 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:18.114739 13022 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.114881 6151 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]). Expected version 1 got 0
I20240503 07:59:18.115026 13022 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:18.115497 6149 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]). Expected version 1 got 0
I20240503 07:59:18.116374 6157 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]). Expected version 1 got 0
I20240503 07:59:18.118690 13234 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.118961 13234 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:18.119370 6157 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb]). Expected version 1 got 0
I20240503 07:59:18.120239 6149 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb] alter complete (version 1)
I20240503 07:59:18.162775 6690 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:18.163158 13114 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.166410 8975 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 07:59:18.168759 9288 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.169405 13080 raft_consensus.cc:1050] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:18.169792 13080 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:18.171257 8575 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:18.171586 8842 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:18.173403 9400 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:18.175107 6430 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:18.175302 9400 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:18.175638 9400 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:18.177157 10079 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.177484 8842 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.177883 6430 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.180075 6157 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.180199 8575 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.183044 8711 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:18.184082 6430 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:18.184325 8842 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:18.184479 13080 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:18.184691 9189 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:18.186107 10079 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.186434 8842 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.187289 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b61209b73f1f4872b5eccc823b1fa3df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.188436 6430 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:18.188874 6149 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.191519 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet b61209b73f1f4872b5eccc823b1fa3df with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:18.191705 13177 tablet_replica.cc:330] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:18.191821 13177 raft_consensus.cc:2229] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.191932 13177 raft_consensus.cc:2258] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.192451 13177 ts_tablet_manager.cc:1875] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.194175 13177 ts_tablet_manager.cc:1888] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:18.194252 13177 log.cc:1196] T b61209b73f1f4872b5eccc823b1fa3df P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/b61209b73f1f4872b5eccc823b1fa3df
I20240503 07:59:18.194674 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet b61209b73f1f4872b5eccc823b1fa3df (table rebalancer_extra_table_8 [id=751d986bca094ff3b81b8f94513d43ec]) successfully deleted
I20240503 07:59:18.222146 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:18.227262 13233 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.227617 13233 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:18.227654 13234 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.227941 13234 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:18.228269 13022 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.228552 13022 tablet.cc:1620] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:18.232303 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=f02e6f3d0e1b40cba8e5569eb31b81cb] alter complete (version 2)
I20240503 07:59:18.237450 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.243196 9288 raft_consensus.cc:1050] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER 616fdda6ba714cb7be01a564de2b9db2 to VOTER
I20240503 07:59:18.243541 9288 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:18.244887 6560 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.245085 6277 tablet_service.cc:1467] Processing CreateTablet for tablet c055763b5be940e9a4ecd63bebd996f8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.245167 8560 tablet_service.cc:1467] Processing CreateTablet for tablet c055763b5be940e9a4ecd63bebd996f8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.245353 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c055763b5be940e9a4ecd63bebd996f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.245316 12434 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.245419 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c055763b5be940e9a4ecd63bebd996f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.246228 8955 tablet_service.cc:1467] Processing CreateTablet for tablet dbee3cf8da48461d94e1ad45ed06a88f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.246492 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbee3cf8da48461d94e1ad45ed06a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.246503 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 3e889be4506e4f01a978f9e94f6b18d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.246742 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e889be4506e4f01a978f9e94f6b18d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.247434 13184 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:18.248190 13182 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:18.248248 13184 tablet_bootstrap.cc:654] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.248368 6670 tablet_service.cc:1467] Processing CreateTablet for tablet dbee3cf8da48461d94e1ad45ed06a88f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.248569 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbee3cf8da48461d94e1ad45ed06a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.249029 13182 tablet_bootstrap.cc:654] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.249732 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 3e889be4506e4f01a978f9e94f6b18d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.249972 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e889be4506e4f01a978f9e94f6b18d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.251749 13183 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:18.252499 13183 tablet_bootstrap.cc:654] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.249747 8822 tablet_service.cc:1467] Processing CreateTablet for tablet c055763b5be940e9a4ecd63bebd996f8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.253053 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c055763b5be940e9a4ecd63bebd996f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.255757 13250 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:18.256065 8711 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.256502 13250 tablet_bootstrap.cc:654] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.256752 13114 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.257050 8690 tablet_service.cc:1467] Processing CreateTablet for tablet dbee3cf8da48461d94e1ad45ed06a88f (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.257277 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 3e889be4506e4f01a978f9e94f6b18d2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.257493 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e889be4506e4f01a978f9e94f6b18d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.257828 8843 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.258040 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbee3cf8da48461d94e1ad45ed06a88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.258139 13114 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.260031 13183 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:18.260103 13183 ts_tablet_manager.cc:1367] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:59:18.260457 13183 raft_consensus.cc:348] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.260542 13183 raft_consensus.cc:374] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.260567 13183 raft_consensus.cc:729] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.260663 13183 consensus_queue.cc:260] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.260921 13183 ts_tablet_manager.cc:1398] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.261075 13183 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:18.261583 13182 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:18.261654 13182 ts_tablet_manager.cc:1367] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:18.261850 13183 tablet_bootstrap.cc:654] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.262055 13182 raft_consensus.cc:348] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.262144 13182 raft_consensus.cc:374] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.262169 13182 raft_consensus.cc:729] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.262264 13182 consensus_queue.cc:260] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.262513 13182 ts_tablet_manager.cc:1398] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.264660 13114 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.264997 8711 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.267164 6560 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.267546 13250 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:18.267618 13250 ts_tablet_manager.cc:1367] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:18.267997 13250 raft_consensus.cc:348] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.268087 13250 raft_consensus.cc:374] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.268117 13250 raft_consensus.cc:729] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.268232 13250 consensus_queue.cc:260] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.268483 13250 ts_tablet_manager.cc:1398] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.268635 13250 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:18.270226 13250 tablet_bootstrap.cc:654] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.271138 13250 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:18.271206 13250 ts_tablet_manager.cc:1367] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.271435 6149 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.271595 13250 raft_consensus.cc:348] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.271678 13250 raft_consensus.cc:374] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.271708 13250 raft_consensus.cc:729] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.271808 13250 consensus_queue.cc:260] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.272047 13250 ts_tablet_manager.cc:1398] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.273675 13259 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:18.274541 13259 tablet_bootstrap.cc:654] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.277051 8843 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.279873 13260 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:18.280755 13260 tablet_bootstrap.cc:654] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.281159 13184 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:18.281270 13184 ts_tablet_manager.cc:1367] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 07:59:18.281739 13184 raft_consensus.cc:348] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.281833 13184 raft_consensus.cc:374] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.281863 13184 raft_consensus.cc:729] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.281973 13184 consensus_queue.cc:260] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.282295 13184 ts_tablet_manager.cc:1398] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.283106 13259 tablet_bootstrap.cc:492] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:18.283228 13259 ts_tablet_manager.cc:1367] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:18.283654 13259 raft_consensus.cc:348] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.283797 13259 raft_consensus.cc:374] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.283869 13259 raft_consensus.cc:729] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.284055 13259 consensus_queue.cc:260] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.284153 13183 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:18.284224 13183 ts_tablet_manager.cc:1367] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.001s
I20240503 07:59:18.284188 6430 consensus_queue.cc:237] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } }
I20240503 07:59:18.284370 13259 ts_tablet_manager.cc:1398] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.284638 13183 raft_consensus.cc:348] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.284719 13183 raft_consensus.cc:374] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.284747 13183 raft_consensus.cc:729] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.284857 13183 consensus_queue.cc:260] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.285112 13183 ts_tablet_manager.cc:1398] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.286146 8843 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.286451 13114 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.287585 8711 raft_consensus.cc:1261] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.288342 13114 consensus_queue.cc:1035] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.288261 13249 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:18.288928 13260 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:18.289072 13260 ts_tablet_manager.cc:1367] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 07:59:18.289434 13249 tablet_bootstrap.cc:654] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.289602 13260 raft_consensus.cc:348] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.289824 13260 raft_consensus.cc:374] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.289899 13260 raft_consensus.cc:729] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.290064 13260 consensus_queue.cc:260] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.290007 13248 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.290412 13260 ts_tablet_manager.cc:1398] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.291121 13249 tablet_bootstrap.cc:492] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:18.291201 13249 ts_tablet_manager.cc:1367] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.291560 13249 raft_consensus.cc:348] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.291716 13249 raft_consensus.cc:374] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.292071 13249 raft_consensus.cc:729] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.292168 8711 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.292819 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edf2c780d0d64db79dca88eac77d4478 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.293247 8843 raft_consensus.cc:2932] T edf2c780d0d64db79dca88eac77d4478 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.295169 6151 catalog_manager.cc:5484] T edf2c780d0d64db79dca88eac77d4478 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } }
I20240503 07:59:18.293110 13249 consensus_queue.cc:260] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.296373 13249 ts_tablet_manager.cc:1398] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:18.298697 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet edf2c780d0d64db79dca88eac77d4478 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:18.299003 13276 tablet_replica.cc:330] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:18.299194 13276 raft_consensus.cc:2229] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.299618 13276 raft_consensus.cc:2258] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.300238 13276 ts_tablet_manager.cc:1875] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.301507 9347 raft_consensus.cc:1050] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:18.301844 9347 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.302326 13276 ts_tablet_manager.cc:1888] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:18.302398 13276 log.cc:1196] T edf2c780d0d64db79dca88eac77d4478 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/edf2c780d0d64db79dca88eac77d4478
I20240503 07:59:18.303079 8843 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.303143 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet edf2c780d0d64db79dca88eac77d4478 (table rebalance_test_table_7 [id=638f2ffce1de4ac6b2869df3c8574ef6]) successfully deleted
I20240503 07:59:18.303428 13175 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.303464 8711 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.303857 6430 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.303856 13175 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.304406 13175 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.306236 9347 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.306962 8843 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.308929 6430 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.309343 6153 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.310613 8711 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.311581 6690 consensus_queue.cc:237] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.312559 8843 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.312985 13175 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.315058 8711 raft_consensus.cc:1261] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.315230 13278 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.315461 12470 consensus_queue.cc:1035] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.316347 8711 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.316865 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 435f73fb9f384b10b353df19037bcac3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.318748 8843 raft_consensus.cc:2932] T 435f73fb9f384b10b353df19037bcac3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.318780 6149 catalog_manager.cc:5484] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 8 to 9, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.321564 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 435f73fb9f384b10b353df19037bcac3 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:18.325590 13287 tablet_replica.cc:330] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:18.325724 13287 raft_consensus.cc:2229] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.325899 13287 raft_consensus.cc:2258] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.327172 13287 ts_tablet_manager.cc:1875] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.330458 13287 ts_tablet_manager.cc:1888] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:18.330521 13287 log.cc:1196] T 435f73fb9f384b10b353df19037bcac3 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/435f73fb9f384b10b353df19037bcac3
I20240503 07:59:18.330945 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 435f73fb9f384b10b353df19037bcac3 (table rebalance_test_table_1 [id=1a03624a2d41492c98d74f5c453afd47]) successfully deleted
I20240503 07:59:18.337551 13278 raft_consensus.cc:482] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.337635 13278 raft_consensus.cc:504] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.337997 13278 leader_election.cc:288] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:18.338157 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:18.338419 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e889be4506e4f01a978f9e94f6b18d2" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:18.338465 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e889be4506e4f01a978f9e94f6b18d2" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:18.338531 6430 raft_consensus.cc:2454] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:18.338569 8843 raft_consensus.cc:2454] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:18.338755 6628 leader_election.cc:302] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:18.338930 13278 raft_consensus.cc:2781] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.338969 13278 raft_consensus.cc:482] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.338999 13278 raft_consensus.cc:3037] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.339807 13278 raft_consensus.cc:504] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.340137 13278 leader_election.cc:288] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:18.340310 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e889be4506e4f01a978f9e94f6b18d2" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:18.340404 8843 raft_consensus.cc:3037] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.341152 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e889be4506e4f01a978f9e94f6b18d2" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:18.341236 6430 raft_consensus.cc:3037] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.341251 8843 raft_consensus.cc:2454] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:18.341542 6627 leader_election.cc:302] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:18.341786 12470 raft_consensus.cc:2781] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.341869 12470 raft_consensus.cc:686] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:18.341977 12470 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:18.342010 6430 raft_consensus.cc:2454] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:18.345794 6153 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.345929 13233 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.346596 13022 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.346583 13234 tablet.cc:1722] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.351488 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=f02e6f3d0e1b40cba8e5569eb31b81cb] alter complete (version 3)
I20240503 07:59:18.353329 13278 raft_consensus.cc:1050] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:18.353685 13278 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.354957 8711 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.354974 6560 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.355265 12470 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.355444 13278 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.355510 8575 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:18.355916 13278 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:18.358036 13278 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.358703 8711 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.359825 6560 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.360697 6149 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.360651 8575 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.364327 6690 consensus_queue.cc:237] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.365443 8575 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.365625 8711 raft_consensus.cc:1261] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 07:59:18.365800 12470 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.365949 9347 consensus_queue.cc:1035] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:18.367635 12470 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.368178 8575 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.369131 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9772b487ca864ad98e9d79d8435c34c6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.370077 8711 raft_consensus.cc:2932] T 9772b487ca864ad98e9d79d8435c34c6 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.370615 6151 catalog_manager.cc:5484] T 9772b487ca864ad98e9d79d8435c34c6 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 8 to 9, VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.372953 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 9772b487ca864ad98e9d79d8435c34c6 with delete_type TABLET_DATA_TOMBSTONED (TS ad910f92ef9d4db8a8962cecba289f8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:18.373207 13276 tablet_replica.cc:330] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:18.373342 13276 raft_consensus.cc:2229] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.373531 13276 raft_consensus.cc:2258] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.374449 13276 ts_tablet_manager.cc:1875] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.376080 13276 ts_tablet_manager.cc:1888] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 07:59:18.376147 13276 log.cc:1196] T 9772b487ca864ad98e9d79d8435c34c6 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9772b487ca864ad98e9d79d8435c34c6
I20240503 07:59:18.376708 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 9772b487ca864ad98e9d79d8435c34c6 (table rebalance_test_table_19 [id=053079d7b58a495aa82a1046fa69c514]) successfully deleted
I20240503 07:59:18.393386 13269 raft_consensus.cc:482] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.393486 13269 raft_consensus.cc:504] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.393766 13269 leader_election.cc:288] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:18.394129 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c055763b5be940e9a4ecd63bebd996f8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:18.394155 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c055763b5be940e9a4ecd63bebd996f8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:18.394245 8575 raft_consensus.cc:2454] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:18.394248 8842 raft_consensus.cc:2454] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:18.394395 12930 raft_consensus.cc:482] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.394493 12930 raft_consensus.cc:504] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.394649 6232 leader_election.cc:302] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:18.394804 13269 raft_consensus.cc:2781] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.394860 13269 raft_consensus.cc:482] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.394845 12930 leader_election.cc:288] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:18.394896 13269 raft_consensus.cc:3037] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.395255 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbee3cf8da48461d94e1ad45ed06a88f" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:18.395255 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbee3cf8da48461d94e1ad45ed06a88f" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:18.395361 8711 raft_consensus.cc:2454] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:18.395361 6690 raft_consensus.cc:2454] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:18.395612 8913 leader_election.cc:302] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:18.395762 12930 raft_consensus.cc:2781] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.395804 12930 raft_consensus.cc:482] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.395834 12930 raft_consensus.cc:3037] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.395893 13269 raft_consensus.cc:504] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.396193 13269 leader_election.cc:288] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:18.396435 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c055763b5be940e9a4ecd63bebd996f8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:18.396548 8842 raft_consensus.cc:3037] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.396677 12930 raft_consensus.cc:504] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.397017 12930 leader_election.cc:288] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:18.397213 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c055763b5be940e9a4ecd63bebd996f8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:18.397213 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbee3cf8da48461d94e1ad45ed06a88f" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:18.397305 8575 raft_consensus.cc:3037] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.397305 8711 raft_consensus.cc:3037] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.397475 8842 raft_consensus.cc:2454] T c055763b5be940e9a4ecd63bebd996f8 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:18.397862 6234 leader_election.cc:302] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:18.397874 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbee3cf8da48461d94e1ad45ed06a88f" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:18.397964 6690 raft_consensus.cc:3037] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.398025 13269 raft_consensus.cc:2781] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.398084 13269 raft_consensus.cc:686] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:18.398101 8711 raft_consensus.cc:2454] T dbee3cf8da48461d94e1ad45ed06a88f P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:18.398101 8575 raft_consensus.cc:2454] T c055763b5be940e9a4ecd63bebd996f8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:18.398192 13269 consensus_queue.cc:237] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.398502 8913 leader_election.cc:302] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:18.398857 6690 raft_consensus.cc:2454] T dbee3cf8da48461d94e1ad45ed06a88f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:18.399552 12930 raft_consensus.cc:2781] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.399648 12930 raft_consensus.cc:686] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:18.399763 12930 consensus_queue.cc:237] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:18.400298 6153 catalog_manager.cc:5484] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.401628 6149 catalog_manager.cc:5484] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.454511 6149 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.454651 6149 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:18.456529 6149 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet ae708f8882794827a03d9fe4a7e6777c
I20240503 07:59:18.457049 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet ae708f8882794827a03d9fe4a7e6777c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:18.457080 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ae708f8882794827a03d9fe4a7e6777c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:18.457219 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet ae708f8882794827a03d9fe4a7e6777c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:18 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:18.457465 13347 tablet_replica.cc:330] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:18.457484 13346 tablet_replica.cc:330] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:18.457593 13348 tablet_replica.cc:330] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:18.457634 13347 raft_consensus.cc:2229] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:18.457659 13346 raft_consensus.cc:2229] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.457712 13348 raft_consensus.cc:2229] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.457819 13348 raft_consensus.cc:2258] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.457819 13347 raft_consensus.cc:2258] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.457808 13346 raft_consensus.cc:2258] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.458348 13348 ts_tablet_manager.cc:1875] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.458348 13347 ts_tablet_manager.cc:1875] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.460039 13346 ts_tablet_manager.cc:1875] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:18.460418 13347 ts_tablet_manager.cc:1888] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.460417 13348 ts_tablet_manager.cc:1888] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.460492 13347 log.cc:1196] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ae708f8882794827a03d9fe4a7e6777c
I20240503 07:59:18.460492 13348 log.cc:1196] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/ae708f8882794827a03d9fe4a7e6777c
I20240503 07:59:18.460810 13348 ts_tablet_manager.cc:1909] T ae708f8882794827a03d9fe4a7e6777c P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:18.460810 13347 ts_tablet_manager.cc:1909] T ae708f8882794827a03d9fe4a7e6777c P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:18.461510 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table_new_name [id=f02e6f3d0e1b40cba8e5569eb31b81cb]) successfully deleted
I20240503 07:59:18.461652 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table_new_name [id=f02e6f3d0e1b40cba8e5569eb31b81cb]) successfully deleted
I20240503 07:59:18.462221 13346 ts_tablet_manager.cc:1888] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:18.462297 13346 log.cc:1196] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae708f8882794827a03d9fe4a7e6777c
I20240503 07:59:18.462558 13346 ts_tablet_manager.cc:1909] T ae708f8882794827a03d9fe4a7e6777c P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:18.463140 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet ae708f8882794827a03d9fe4a7e6777c (table rebalancer_dynamic_table_new_name [id=f02e6f3d0e1b40cba8e5569eb31b81cb]) successfully deleted
I20240503 07:59:18.467025 12608 raft_consensus.cc:1050] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 to VOTER
I20240503 07:59:18.467413 12608 consensus_queue.cc:237] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.468626 6298 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.468716 8711 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.468794 6428 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:18.469051 12344 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.469193 13017 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.469379 13018 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.471115 13018 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.471727 8711 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.472903 6298 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.472949 6428 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.475719 6149 catalog_manager.cc:5484] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 4 to 5, 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.479472 6560 consensus_queue.cc:237] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } }
I20240503 07:59:18.480455 8711 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:18.480804 12344 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:18.480928 6298 raft_consensus.cc:1261] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:18.481287 12344 consensus_queue.cc:1035] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:18.482115 13018 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.483690 6298 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.484186 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afcd161598db41d7962caca61f63ef77 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.484405 8711 raft_consensus.cc:2932] T afcd161598db41d7962caca61f63ef77 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } } }
I20240503 07:59:18.485417 6151 catalog_manager.cc:5484] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 5 to 6, VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.487784 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet afcd161598db41d7962caca61f63ef77 with delete_type TABLET_DATA_TOMBSTONED (TS d1f36d57661b4429a4be8d10b58d8ff7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:18.487980 13287 tablet_replica.cc:330] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:18.488111 13287 raft_consensus.cc:2229] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.488256 13287 raft_consensus.cc:2258] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.488991 13287 ts_tablet_manager.cc:1875] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.490597 13287 ts_tablet_manager.cc:1888] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 07:59:18.490666 13287 log.cc:1196] T afcd161598db41d7962caca61f63ef77 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/afcd161598db41d7962caca61f63ef77
I20240503 07:59:18.491134 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet afcd161598db41d7962caca61f63ef77 (table rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]) successfully deleted
I20240503 07:59:18.558351 6153 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:18.562973 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 65780fa05abc434a880cdb395b3053b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]), partition=
I20240503 07:59:18.563174 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 65780fa05abc434a880cdb395b3053b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]), partition=
I20240503 07:59:18.563230 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65780fa05abc434a880cdb395b3053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.563377 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65780fa05abc434a880cdb395b3053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.564895 13182 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:18.565061 13183 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:18.565721 13182 tablet_bootstrap.cc:654] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.565825 13183 tablet_bootstrap.cc:654] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.566391 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 65780fa05abc434a880cdb395b3053b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]), partition=
I20240503 07:59:18.566592 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65780fa05abc434a880cdb395b3053b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.566733 13182 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:18.566814 13182 ts_tablet_manager.cc:1367] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.567224 13182 raft_consensus.cc:348] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.567348 13182 raft_consensus.cc:374] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.567392 13182 raft_consensus.cc:729] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.567513 13182 consensus_queue.cc:260] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.567790 13182 ts_tablet_manager.cc:1398] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:18.568497 13249 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:18.569365 13249 tablet_bootstrap.cc:654] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.570156 13183 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:18.570226 13183 ts_tablet_manager.cc:1367] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:18.570619 13183 raft_consensus.cc:348] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.570709 13183 raft_consensus.cc:374] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.570735 13183 raft_consensus.cc:729] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.570843 13183 consensus_queue.cc:260] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.571101 13183 ts_tablet_manager.cc:1398] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.571659 13249 tablet_bootstrap.cc:492] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:18.571728 13249 ts_tablet_manager.cc:1367] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.572484 13249 raft_consensus.cc:348] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.572587 13249 raft_consensus.cc:374] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.572623 13249 raft_consensus.cc:729] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.572727 13249 consensus_queue.cc:260] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.573016 13249 ts_tablet_manager.cc:1398] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.586771 6560 consensus_queue.cc:237] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:18.587814 8842 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:18.588068 6690 raft_consensus.cc:1261] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:18.588340 12344 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.588666 12344 consensus_queue.cc:1035] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:18.590636 13017 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:18.591051 8842 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:18.591048 6690 raft_consensus.cc:2932] T 09866b93f75545a19c9451556254da48 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:18.592300 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09866b93f75545a19c9451556254da48 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.593467 6153 catalog_manager.cc:5484] T 09866b93f75545a19c9451556254da48 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 3 to 5, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.595530 8690 tablet_service.cc:1514] Processing DeleteTablet for tablet 09866b93f75545a19c9451556254da48 with delete_type TABLET_DATA_TOMBSTONED (TS 9553fe3c4ccd4e91b7c176aad82fec09 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:18.596226 13356 tablet_replica.cc:330] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:18.596352 13356 raft_consensus.cc:2229] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.596462 13356 raft_consensus.cc:2258] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.601169 13356 ts_tablet_manager.cc:1875] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.603080 13356 ts_tablet_manager.cc:1888] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 07:59:18.603142 13356 log.cc:1196] T 09866b93f75545a19c9451556254da48 P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/09866b93f75545a19c9451556254da48
I20240503 07:59:18.603528 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 09866b93f75545a19c9451556254da48 (table rebalancer_extra_table_4 [id=cc7099c0515845cba9ea697203082da7]) successfully deleted
I20240503 07:59:18.608464 13180 raft_consensus.cc:482] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.608577 13180 raft_consensus.cc:504] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.608976 13180 leader_election.cc:288] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:18.609227 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65780fa05abc434a880cdb395b3053b0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:18.609314 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65780fa05abc434a880cdb395b3053b0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:18.609355 8579 raft_consensus.cc:2454] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:18.609408 8842 raft_consensus.cc:2454] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:18.609663 8910 leader_election.cc:302] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:18.609820 13180 raft_consensus.cc:2781] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.609882 13180 raft_consensus.cc:482] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.609917 13180 raft_consensus.cc:3037] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.610811 13180 raft_consensus.cc:504] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.611115 13180 leader_election.cc:288] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:18.611613 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65780fa05abc434a880cdb395b3053b0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:18.611716 8842 raft_consensus.cc:3037] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.612608 8842 raft_consensus.cc:2454] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:18.612918 8912 leader_election.cc:302] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:18.613063 13180 raft_consensus.cc:2781] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.613145 13180 raft_consensus.cc:686] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:18.613248 13180 consensus_queue.cc:237] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:18.617072 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65780fa05abc434a880cdb395b3053b0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:18.617187 8579 raft_consensus.cc:3037] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.617996 6153 catalog_manager.cc:5484] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:18.618726 8579 raft_consensus.cc:2454] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:18.729074 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:18.732645 8579 raft_consensus.cc:1261] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.732719 8842 raft_consensus.cc:1261] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:18.733104 12930 consensus_queue.cc:1035] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.733390 12930 consensus_queue.cc:1035] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.735388 13233 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.735677 13233 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:18.735816 6153 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]). Expected version 1 got 0
I20240503 07:59:18.736337 13022 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.736490 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]). Expected version 1 got 0
I20240503 07:59:18.736589 13022 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:18.737726 6153 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c]). Expected version 1 got 0
I20240503 07:59:18.739787 6149 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c] alter complete (version 1)
I20240503 07:59:18.741158 13381 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:18.741429 13381 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:18.762696 13278 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.766427 9347 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.774998 6428 consensus_queue.cc:237] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:18.776074 8579 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:18.776098 8975 raft_consensus.cc:1261] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 07:59:18.776425 9288 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:18.776638 13248 consensus_queue.cc:1035] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:18.778210 13114 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:18.778649 8579 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:18.778645 8975 raft_consensus.cc:2932] T 8861b52c306e43d8b0615202cf7d8e35 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } }
I20240503 07:59:18.779726 6144 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8861b52c306e43d8b0615202cf7d8e35 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:18.781255 6149 catalog_manager.cc:5484] T 8861b52c306e43d8b0615202cf7d8e35 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 8 to 10, VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.783370 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 8861b52c306e43d8b0615202cf7d8e35 with delete_type TABLET_DATA_TOMBSTONED (TS a97073f89fd44702bccb74a9b76f858d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:18.783906 13386 tablet_replica.cc:330] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:18.784019 13386 raft_consensus.cc:2229] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:18.784098 13386 raft_consensus.cc:2258] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:18.784679 13386 ts_tablet_manager.cc:1875] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:18.786652 13386 ts_tablet_manager.cc:1888] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 07:59:18.786715 13386 log.cc:1196] T 8861b52c306e43d8b0615202cf7d8e35 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8861b52c306e43d8b0615202cf7d8e35
I20240503 07:59:18.787127 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 8861b52c306e43d8b0615202cf7d8e35 (table rebalance_test_table_3 [id=2040f6f2be794bbea4b293e02e8ee6e1]) successfully deleted
I20240503 07:59:18.805429 13180 consensus_queue.cc:1035] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.808355 13180 consensus_queue.cc:1035] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.816519 13034 consensus_queue.cc:1035] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.819545 13034 consensus_queue.cc:1035] T c055763b5be940e9a4ecd63bebd996f8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:18.843573 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:18.848989 13233 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.849310 13233 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:18.849737 13022 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.850003 13022 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:18.850889 13381 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:18.851150 13381 tablet.cc:1620] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:18.855221 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=657ed53b726f4fcbad91b81e50cc639c] alter complete (version 2)
I20240503 07:59:18.910691 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:18.918125 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.918442 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.919070 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 3488a545a7d84e389c5894d9ba3a592b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.919308 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3488a545a7d84e389c5894d9ba3a592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.919658 8692 tablet_service.cc:1467] Processing CreateTablet for tablet 3488a545a7d84e389c5894d9ba3a592b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.919873 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3488a545a7d84e389c5894d9ba3a592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.920724 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.920969 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.921384 13435 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:18.921802 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 856c66722df446d1ae96fe248bc31941 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.922049 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856c66722df446d1ae96fe248bc31941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.922544 13435 tablet_bootstrap.cc:654] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.922923 8690 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.923138 8690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e6eef6b9c8548c3a790d42a6a9eef4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.924185 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 3488a545a7d84e389c5894d9ba3a592b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.924412 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3488a545a7d84e389c5894d9ba3a592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.925127 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 856c66722df446d1ae96fe248bc31941 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.925329 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856c66722df446d1ae96fe248bc31941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.926951 13249 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:18.927098 13182 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:18.927989 13182 tablet_bootstrap.cc:654] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.928695 13249 tablet_bootstrap.cc:654] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.929033 13182 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:18.929107 13182 ts_tablet_manager.cc:1367] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:18.929656 13182 raft_consensus.cc:348] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.929757 13249 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:18.929761 13182 raft_consensus.cc:374] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.929793 13182 raft_consensus.cc:729] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.929824 13249 ts_tablet_manager.cc:1367] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:18.929917 13182 consensus_queue.cc:260] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.930193 13182 ts_tablet_manager.cc:1398] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.930222 13249 raft_consensus.cc:348] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.930321 13249 raft_consensus.cc:374] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.930361 13249 raft_consensus.cc:729] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.930469 13249 consensus_queue.cc:260] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.930519 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 856c66722df446d1ae96fe248bc31941 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=499ce2d4358448c3aea77c2a544c097c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:18.930716 13249 ts_tablet_manager.cc:1398] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:18.930757 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856c66722df446d1ae96fe248bc31941. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:18.933151 13183 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:18.933966 13183 tablet_bootstrap.cc:654] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.935679 13438 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:18.937458 13438 tablet_bootstrap.cc:654] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.937490 13183 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:18.937561 13183 ts_tablet_manager.cc:1367] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:18.937964 13183 raft_consensus.cc:348] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.938051 13183 raft_consensus.cc:374] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.938079 13183 raft_consensus.cc:729] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.938189 13183 consensus_queue.cc:260] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.938460 13183 ts_tablet_manager.cc:1398] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.938589 13438 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:18.938658 13438 ts_tablet_manager.cc:1367] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:18.939044 13438 raft_consensus.cc:348] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.939162 13438 raft_consensus.cc:374] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.939209 13438 raft_consensus.cc:729] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.940156 13435 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:18.940290 13435 ts_tablet_manager.cc:1367] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:18.940479 13436 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:18.939328 13438 consensus_queue.cc:260] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.940930 13435 raft_consensus.cc:348] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.941067 13435 raft_consensus.cc:374] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.941138 13435 raft_consensus.cc:729] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.941219 13438 ts_tablet_manager.cc:1398] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.941318 13435 consensus_queue.cc:260] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.942832 13436 tablet_bootstrap.cc:654] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.943526 13435 ts_tablet_manager.cc:1398] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.943797 13435 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:18.943941 13436 tablet_bootstrap.cc:492] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:18.944038 13436 ts_tablet_manager.cc:1367] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.944445 13436 raft_consensus.cc:348] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.944545 13436 raft_consensus.cc:374] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.944581 13436 raft_consensus.cc:729] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.944653 13435 tablet_bootstrap.cc:654] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.944692 13436 consensus_queue.cc:260] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.945008 13436 ts_tablet_manager.cc:1398] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.945187 13436 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:18.945940 13436 tablet_bootstrap.cc:654] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.947918 13435 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:18.948014 13435 ts_tablet_manager.cc:1367] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:18.950605 13435 raft_consensus.cc:348] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.950735 13435 raft_consensus.cc:374] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.950819 13435 raft_consensus.cc:729] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.950989 13435 consensus_queue.cc:260] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.951249 13435 ts_tablet_manager.cc:1398] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:18.951898 13436 tablet_bootstrap.cc:492] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:18.952004 13436 ts_tablet_manager.cc:1367] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:18.952327 13436 raft_consensus.cc:348] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.952472 13436 raft_consensus.cc:374] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.952519 13436 raft_consensus.cc:729] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.953300 13436 consensus_queue.cc:260] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.953619 13436 ts_tablet_manager.cc:1398] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:18.954123 13436 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:18.956490 13436 tablet_bootstrap.cc:654] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:18.958561 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:18.963689 12434 raft_consensus.cc:482] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.963780 12434 raft_consensus.cc:504] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.964155 12434 leader_election.cc:288] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:18.964819 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3488a545a7d84e389c5894d9ba3a592b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:18.964946 8711 raft_consensus.cc:2454] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:18.965047 13233 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.965176 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3488a545a7d84e389c5894d9ba3a592b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:18.965243 6366 leader_election.cc:302] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:18.965288 6560 raft_consensus.cc:2454] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:18.965458 12434 raft_consensus.cc:2781] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.965520 12434 raft_consensus.cc:482] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.965559 12434 raft_consensus.cc:3037] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.966629 12434 raft_consensus.cc:504] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.966903 13381 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.967001 12434 leader_election.cc:288] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:18.967427 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3488a545a7d84e389c5894d9ba3a592b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:18.967526 6560 raft_consensus.cc:3037] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.968523 6560 raft_consensus.cc:2454] T 3488a545a7d84e389c5894d9ba3a592b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:18.968572 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3488a545a7d84e389c5894d9ba3a592b" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:18.968668 8711 raft_consensus.cc:3037] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.968834 6363 leader_election.cc:302] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:18.968999 12434 raft_consensus.cc:2781] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.969079 12434 raft_consensus.cc:686] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:18.969204 12434 consensus_queue.cc:237] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:18.969564 8711 raft_consensus.cc:2454] T 3488a545a7d84e389c5894d9ba3a592b P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:18.969630 13022 tablet.cc:1722] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:18.969707 12608 raft_consensus.cc:482] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:18.969774 12608 raft_consensus.cc:504] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.970094 12608 leader_election.cc:288] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:18.971659 6150 catalog_manager.cc:5484] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.972157 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6eef6b9c8548c3a790d42a6a9eef4c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:18.972259 8711 raft_consensus.cc:2454] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:18.972534 6497 leader_election.cc:302] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:18.972936 12608 raft_consensus.cc:2781] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:18.972999 12608 raft_consensus.cc:482] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:18.973039 12608 raft_consensus.cc:3037] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.973028 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6eef6b9c8548c3a790d42a6a9eef4c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:18.973083 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=657ed53b726f4fcbad91b81e50cc639c] alter complete (version 3)
I20240503 07:59:18.973145 8975 raft_consensus.cc:2454] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:18.974916 12608 raft_consensus.cc:504] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.975396 12608 leader_election.cc:288] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:18.975943 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6eef6b9c8548c3a790d42a6a9eef4c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:18.976059 8975 raft_consensus.cc:3037] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.976984 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e6eef6b9c8548c3a790d42a6a9eef4c" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:18.977018 8975 raft_consensus.cc:2454] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:18.977074 8711 raft_consensus.cc:3037] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:18.977921 8711 raft_consensus.cc:2454] T 3e6eef6b9c8548c3a790d42a6a9eef4c P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:18.978205 6497 leader_election.cc:302] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9553fe3c4ccd4e91b7c176aad82fec09, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:18.978540 12344 raft_consensus.cc:2781] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:18.978621 12344 raft_consensus.cc:686] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:18.978749 12344 consensus_queue.cc:237] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:18.981040 13436 tablet_bootstrap.cc:492] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:18.981133 13436 ts_tablet_manager.cc:1367] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 07:59:18.981596 13436 raft_consensus.cc:348] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.981689 13436 raft_consensus.cc:374] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:18.981721 13436 raft_consensus.cc:729] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:18.981848 13436 consensus_queue.cc:260] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:18.982138 13436 ts_tablet_manager.cc:1398] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:18.982156 6157 catalog_manager.cc:5484] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:18.985149 8822 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:18.985536 6670 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 07:59:18.985855 8691 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 07:59:19.000393 8955 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:19.002403 6539 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 07:59:19.006804 6277 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:19.013106 6408 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:19.035219 8560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:19.078907 6157 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:19.079044 6157 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:19.081256 6157 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 65780fa05abc434a880cdb395b3053b0
I20240503 07:59:19.081919 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 65780fa05abc434a880cdb395b3053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:19.082151 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 65780fa05abc434a880cdb395b3053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:19.082237 13471 tablet_replica.cc:330] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:19.082252 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet 65780fa05abc434a880cdb395b3053b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:19.082377 13471 raft_consensus.cc:2229] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.082535 13471 raft_consensus.cc:2258] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.083045 13473 tablet_replica.cc:330] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:19.083195 13473 raft_consensus.cc:2229] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:19.083191 13471 ts_tablet_manager.cc:1875] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.083403 13473 raft_consensus.cc:2258] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.082688 13472 tablet_replica.cc:330] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:19.083868 13472 raft_consensus.cc:2229] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.084057 13472 raft_consensus.cc:2258] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.085166 13473 ts_tablet_manager.cc:1875] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.086066 13471 ts_tablet_manager.cc:1888] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.086143 13471 log.cc:1196] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/65780fa05abc434a880cdb395b3053b0
I20240503 07:59:19.086275 8842 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.086388 13471 ts_tablet_manager.cc:1909] T 65780fa05abc434a880cdb395b3053b0 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:19.087128 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table_new_name [id=657ed53b726f4fcbad91b81e50cc639c]) successfully deleted
I20240503 07:59:19.087194 13473 ts_tablet_manager.cc:1888] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.087255 13473 log.cc:1196] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/65780fa05abc434a880cdb395b3053b0
I20240503 07:59:19.087503 13473 ts_tablet_manager.cc:1909] T 65780fa05abc434a880cdb395b3053b0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:19.087677 8579 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.087776 12241 raft_consensus.cc:482] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.087855 12241 raft_consensus.cc:504] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:19.088157 12241 leader_election.cc:288] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.088194 13472 ts_tablet_manager.cc:1875] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.088176 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table_new_name [id=657ed53b726f4fcbad91b81e50cc639c]) successfully deleted
W20240503 07:59:19.088575 8776 consensus_peers.cc:485] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b745ba777b2476fbae78706b35125f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.088671 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856c66722df446d1ae96fe248bc31941" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:19.088824 8579 raft_consensus.cc:2454] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:19.088977 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856c66722df446d1ae96fe248bc31941" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:19.089025 6428 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.089103 8711 raft_consensus.cc:2454] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:19.089421 8776 leader_election.cc:302] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.089395 11508 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.089896 11508 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.090126 13472 ts_tablet_manager.cc:1888] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.090160 11508 raft_consensus.cc:2781] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.090196 11508 raft_consensus.cc:482] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.090278 11508 raft_consensus.cc:3037] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.091109 11508 raft_consensus.cc:504] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:19.091437 11508 leader_election.cc:288] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.091694 8711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856c66722df446d1ae96fe248bc31941" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:19.091802 8711 raft_consensus.cc:3037] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.091884 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856c66722df446d1ae96fe248bc31941" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:19.091966 8579 raft_consensus.cc:3037] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.091958 11508 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.092414 6428 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.092725 8579 raft_consensus.cc:2454] T 856c66722df446d1ae96fe248bc31941 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:19.092725 8711 raft_consensus.cc:2454] T 856c66722df446d1ae96fe248bc31941 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:19.093008 8779 leader_election.cc:302] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, 9553fe3c4ccd4e91b7c176aad82fec09; no voters:
I20240503 07:59:19.093185 11508 raft_consensus.cc:2781] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.093254 11508 raft_consensus.cc:686] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:19.093406 11508 consensus_queue.cc:237] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:19.093873 8579 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.090194 13472 log.cc:1196] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/65780fa05abc434a880cdb395b3053b0
I20240503 07:59:19.094579 13472 ts_tablet_manager.cc:1909] T 65780fa05abc434a880cdb395b3053b0 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:19.094590 6157 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.095304 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 65780fa05abc434a880cdb395b3053b0 (table rebalancer_dynamic_table_new_name [id=657ed53b726f4fcbad91b81e50cc639c]) successfully deleted
I20240503 07:59:19.098680 6150 catalog_manager.cc:5484] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.105577 6560 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.106763 8975 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.106801 8842 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.106959 6496 consensus_peers.cc:485] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edaf4aa5285f49c297ab412ceb179be9. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.107153 13018 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.107306 12344 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.108809 12608 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.109251 8842 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.109335 8975 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.111743 6151 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.119647 6428 consensus_queue.cc:237] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } }
I20240503 07:59:19.120735 8579 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:19.120815 6560 raft_consensus.cc:1261] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:19.120944 6366 consensus_peers.cc:485] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60f643dce3eb4c669b36b0714a31651b. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.121130 12434 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:19.121294 9288 consensus_queue.cc:1035] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:19.122812 13248 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:19.124289 6560 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:19.124289 8579 raft_consensus.cc:2932] T 60f643dce3eb4c669b36b0714a31651b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } } }
I20240503 07:59:19.126681 6157 catalog_manager.cc:5484] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: NON_VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.133930 8975 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.135133 8842 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.135185 8579 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.135350 8913 consensus_peers.cc:485] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1b2ea716ca64e309efccd291db503d6. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.135550 12930 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.135707 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.137159 13180 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.137549 8842 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.138520 8579 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.139961 6150 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.146919 8711 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.148129 6428 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:19.148514 13510 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:19.150002 13509 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.150398 6428 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.152998 8842 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:19.153097 6157 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:19.153316 8647 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b61209b73f1f4872b5eccc823b1fa3df. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.153496 13510 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:19.154273 8842 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.162286 6560 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.163607 8975 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.163619 6494 consensus_peers.cc:485] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7442d12e0234db38d896c09a9e73162. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.163646 6690 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.163941 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.164228 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.165549 13018 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.165956 8975 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.168864 6151 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.171080 6690 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.182432 8842 consensus_queue.cc:237] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.182873 6157 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:19.183774 6560 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.183836 8711 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.184007 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfd4deb6a5104141bfa51fe4114bb708. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.184190 13475 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.184351 12241 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.186054 13475 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.187306 6560 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.187438 8691 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe511d140c8487b9ede11a841d48bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]), partition=
I20240503 07:59:19.187656 8691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe511d140c8487b9ede11a841d48bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.187772 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe511d140c8487b9ede11a841d48bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]), partition=
I20240503 07:59:19.187963 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe511d140c8487b9ede11a841d48bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.188088 8711 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.188725 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe511d140c8487b9ede11a841d48bba (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]), partition=
I20240503 07:59:19.188992 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe511d140c8487b9ede11a841d48bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.189913 13249 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:19.190215 13436 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Bootstrap starting.
I20240503 07:59:19.191299 13249 tablet_bootstrap.cc:654] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.191313 13436 tablet_bootstrap.cc:654] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.192415 6151 catalog_manager.cc:5484] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.192922 13527 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:19.194020 13527 tablet_bootstrap.cc:654] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.194314 13436 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: No bootstrap required, opened a new log
I20240503 07:59:19.194392 13436 ts_tablet_manager.cc:1367] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:19.194499 13249 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:19.194564 13249 ts_tablet_manager.cc:1367] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240503 07:59:19.194808 13436 raft_consensus.cc:348] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.194890 13436 raft_consensus.cc:374] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.194919 13436 raft_consensus.cc:729] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553fe3c4ccd4e91b7c176aad82fec09, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.194933 13249 raft_consensus.cc:348] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.195009 13249 raft_consensus.cc:374] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.195036 13249 raft_consensus.cc:729] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.195022 13436 consensus_queue.cc:260] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.195144 13249 consensus_queue.cc:260] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.195279 13436 ts_tablet_manager.cc:1398] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.196977 13249 ts_tablet_manager.cc:1398] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.199236 13527 tablet_bootstrap.cc:492] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:19.199323 13527 ts_tablet_manager.cc:1367] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:19.199739 13527 raft_consensus.cc:348] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.199834 13527 raft_consensus.cc:374] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.199889 13527 raft_consensus.cc:729] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.200011 13527 consensus_queue.cc:260] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.200271 13527 ts_tablet_manager.cc:1398] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.204866 6296 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.206024 6428 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 07:59:19.206043 8711 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 07:59:19.206288 6234 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551f25dcee034b7894498e3c748b0f0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.206409 13537 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:19.206576 13269 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:19.208201 13034 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.208604 6428 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.209797 13034 raft_consensus.cc:482] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.209869 13034 raft_consensus.cc:504] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.210001 8711 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.210157 13034 leader_election.cc:288] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:19.210464 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe511d140c8487b9ede11a841d48bba" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:19.210566 8579 raft_consensus.cc:2454] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:19.210816 6232 leader_election.cc:302] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.211135 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe511d140c8487b9ede11a841d48bba" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" is_pre_election: true
I20240503 07:59:19.211163 13537 raft_consensus.cc:2781] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.211205 13537 raft_consensus.cc:482] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.211237 13537 raft_consensus.cc:3037] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.211230 8712 raft_consensus.cc:2454] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:19.211421 6157 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 4 to 5, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.212061 13537 raft_consensus.cc:504] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.212358 13537 leader_election.cc:288] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:19.212570 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe511d140c8487b9ede11a841d48bba" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:19.212662 8579 raft_consensus.cc:3037] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.212641 8712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe511d140c8487b9ede11a841d48bba" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553fe3c4ccd4e91b7c176aad82fec09"
I20240503 07:59:19.212731 8712 raft_consensus.cc:3037] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.213507 8712 raft_consensus.cc:2454] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:19.213519 8579 raft_consensus.cc:2454] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:19.214037 6232 leader_election.cc:302] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.216365 13537 raft_consensus.cc:2781] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.216495 13537 raft_consensus.cc:686] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:19.216639 13537 consensus_queue.cc:237] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:19.218294 6151 catalog_manager.cc:5484] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:19.221874 6690 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:19.223140 8842 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.223235 6428 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.223250 6625 consensus_peers.cc:485] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e889be4506e4f01a978f9e94f6b18d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.223575 13175 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.223722 9347 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.225064 13175 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.225438 8842 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.225669 6428 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.227576 6151 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.235211 8712 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.236481 8579 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.236521 8645 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00bcbd5440ff4d56b1975edc4041b428. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.236565 8975 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 9553fe3c4ccd4e91b7c176aad82fec09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.236914 9400 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.237067 13510 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.238801 9400 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.240749 8579 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.240983 8975 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.241365 6150 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09 reported cstate change: config changed from index -1 to 2, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.252055 8842 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.253428 8975 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.253453 8712 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.253805 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc7d26466129405bbe6db2721823bae1. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.253811 13475 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.254000 13478 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.255625 11508 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.256556 8712 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.257186 8976 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } } }
I20240503 07:59:19.258442 6153 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.268018 8579 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.269371 8842 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.269498 6428 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.269846 8516 consensus_peers.cc:485] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa8b855a6985404da6bd34daa0eb9ae7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.269865 13568 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.270288 13392 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.271531 13568 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.271927 8842 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.272245 6428 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.274817 6150 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index -1 to 2, NON_VOTER a97073f89fd44702bccb74a9b76f858d (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } } }
I20240503 07:59:19.282286 6690 consensus_queue.cc:237] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:19.283421 8976 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.283443 8712 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 07:59:19.283596 6625 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): Couldn't send request to peer ad910f92ef9d4db8a8962cecba289f8a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2054e7790d7745c2a1b08e903c64d1b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.283809 12470 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.284054 13278 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:19.285483 9347 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.285873 8976 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.286109 8712 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.289000 6151 catalog_manager.cc:5484] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index -1 to 2, NON_VOTER ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } } }
I20240503 07:59:19.295591 6428 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.296957 8976 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 07:59:19.297122 6363 consensus_peers.cc:485] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c01e3ce57874e24ae75a41243fdeaaf. This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.297369 6560 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:19.297442 12434 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:19.297819 12434 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:19.298956 13248 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.299347 8976 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.301714 6153 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 6 to 7, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.301800 6560 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c167a0a433d34b74affed13cbf96e302 (127.0.99.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } } }
I20240503 07:59:19.322569 6153 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:19.326886 8579 raft_consensus.cc:1261] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.327014 8712 raft_consensus.cc:1261] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.327420 13034 consensus_queue.cc:1035] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.329452 13034 consensus_queue.cc:1035] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.330634 6153 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]). Expected version 1 got 0
I20240503 07:59:19.333513 13617 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.333923 13617 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:19.335434 6150 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]). Expected version 1 got 0
I20240503 07:59:19.335840 13381 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.336117 13381 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:19.336194 6153 catalog_manager.cc:5576] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925) does not have the latest schema for tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]). Expected version 1 got 0
I20240503 07:59:19.337095 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]). Expected version 1 got 0
I20240503 07:59:19.339906 6150 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]). Expected version 1 got 0
I20240503 07:59:19.340633 6153 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea] alter complete (version 1)
I20240503 07:59:19.345201 13615 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.345471 13615 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:19.382809 13018 consensus_queue.cc:1035] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.386797 12608 consensus_queue.cc:1035] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.388808 13114 consensus_queue.cc:1035] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.393096 13248 consensus_queue.cc:1035] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.442677 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:19.447603 13615 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.447901 13615 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:19.448053 13617 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.448302 13617 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: MemRowSet was empty: no flush needed.
I20240503 07:59:19.451154 13381 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:19.451428 13381 tablet.cc:1620] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:19.452963 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c1422f5b84ef4bdfb6a9e0a5ed147fea] alter complete (version 2)
I20240503 07:59:19.504315 13669 ts_tablet_manager.cc:902] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:19.504810 13669 tablet_copy_client.cc:308] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:19.505218 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet edaf4aa5285f49c297ab412ceb179be9 from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:41925)
I20240503 07:59:19.505328 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet edaf4aa5285f49c297ab412ceb179be9 from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:41925: session id = a97073f89fd44702bccb74a9b76f858d-edaf4aa5285f49c297ab412ceb179be9
I20240503 07:59:19.505991 6581 tablet_copy_source_session.cc:205] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.506784 13669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf4aa5285f49c297ab412ceb179be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.508538 13669 tablet_copy_client.cc:783] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.508668 13669 tablet_copy_client.cc:647] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.509742 13669 tablet_copy_client.cc:523] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.510680 13669 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:19.513316 13669 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.513494 13669 tablet_bootstrap.cc:492] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:19.513605 13669 ts_tablet_manager.cc:1367] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:19.514405 13669 raft_consensus.cc:348] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.514511 13669 raft_consensus.cc:729] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:19.514586 13669 consensus_queue.cc:260] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.514811 13669 ts_tablet_manager.cc:1398] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.515225 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-edaf4aa5285f49c297ab412ceb179be9 received from {username='slave'} at 127.0.99.132:41925
I20240503 07:59:19.515305 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-edaf4aa5285f49c297ab412ceb179be9 on tablet edaf4aa5285f49c297ab412ceb179be9 with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:19.526759 13475 consensus_queue.cc:1035] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.530248 13478 consensus_queue.cc:1035] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:19.530623 13677 ts_tablet_manager.cc:902] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Initiating tablet copy from peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:19.531112 13677 tablet_copy_client.cc:308] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:39417
I20240503 07:59:19.533059 8864 tablet_copy_service.cc:140] P 616fdda6ba714cb7be01a564de2b9db2: Received BeginTabletCopySession request for tablet 5b745ba777b2476fbae78706b35125f6 from peer c167a0a433d34b74affed13cbf96e302 ({username='slave'} at 127.0.99.129:36451)
I20240503 07:59:19.533149 8864 tablet_copy_service.cc:161] P 616fdda6ba714cb7be01a564de2b9db2: Beginning new tablet copy session on tablet 5b745ba777b2476fbae78706b35125f6 from peer c167a0a433d34b74affed13cbf96e302 at {username='slave'} at 127.0.99.129:36451: session id = c167a0a433d34b74affed13cbf96e302-5b745ba777b2476fbae78706b35125f6
I20240503 07:59:19.534063 8864 tablet_copy_source_session.cc:205] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.534843 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b745ba777b2476fbae78706b35125f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.537200 13677 tablet_copy_client.cc:783] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.537389 13677 tablet_copy_client.cc:647] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.538528 13677 tablet_copy_client.cc:523] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.539402 13677 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:19.541972 13677 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.542138 13677 tablet_bootstrap.cc:492] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Bootstrap complete.
I20240503 07:59:19.542244 13677 ts_tablet_manager.cc:1367] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:19.542557 13677 raft_consensus.cc:348] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.542651 13677 raft_consensus.cc:729] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: LEARNER
I20240503 07:59:19.542742 13677 consensus_queue.cc:260] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.542992 13677 ts_tablet_manager.cc:1398] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.543385 8864 tablet_copy_service.cc:342] P 616fdda6ba714cb7be01a564de2b9db2: Request end of tablet copy session c167a0a433d34b74affed13cbf96e302-5b745ba777b2476fbae78706b35125f6 received from {username='slave'} at 127.0.99.129:36451
I20240503 07:59:19.543462 8864 tablet_copy_service.cc:433] P 616fdda6ba714cb7be01a564de2b9db2: ending tablet copy session c167a0a433d34b74affed13cbf96e302-5b745ba777b2476fbae78706b35125f6 on tablet 5b745ba777b2476fbae78706b35125f6 with peer c167a0a433d34b74affed13cbf96e302
I20240503 07:59:19.555294 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:19.560359 13615 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.560891 13617 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.562013 13381 tablet.cc:1722] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:19.565553 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=c1422f5b84ef4bdfb6a9e0a5ed147fea] alter complete (version 3)
I20240503 07:59:19.586409 13684 ts_tablet_manager.cc:902] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:19.586860 13684 tablet_copy_client.cc:308] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:41089
I20240503 07:59:19.587266 8997 tablet_copy_service.cc:140] P 3d08a06d53b14381bee6223c4e1a6027: Received BeginTabletCopySession request for tablet e1b2ea716ca64e309efccd291db503d6 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:56247)
I20240503 07:59:19.587360 8997 tablet_copy_service.cc:161] P 3d08a06d53b14381bee6223c4e1a6027: Beginning new tablet copy session on tablet e1b2ea716ca64e309efccd291db503d6 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:56247: session id = d1f36d57661b4429a4be8d10b58d8ff7-e1b2ea716ca64e309efccd291db503d6
I20240503 07:59:19.588204 8997 tablet_copy_source_session.cc:205] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.588337 13677 ts_tablet_manager.cc:902] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Initiating tablet copy from peer ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:19.588728 13677 tablet_copy_client.cc:308] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:45693
I20240503 07:59:19.589069 13684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b2ea716ca64e309efccd291db503d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.589164 6581 tablet_copy_service.cc:140] P ad910f92ef9d4db8a8962cecba289f8a: Received BeginTabletCopySession request for tablet f7442d12e0234db38d896c09a9e73162 from peer c167a0a433d34b74affed13cbf96e302 ({username='slave'} at 127.0.99.129:58657)
I20240503 07:59:19.589259 6581 tablet_copy_service.cc:161] P ad910f92ef9d4db8a8962cecba289f8a: Beginning new tablet copy session on tablet f7442d12e0234db38d896c09a9e73162 from peer c167a0a433d34b74affed13cbf96e302 at {username='slave'} at 127.0.99.129:58657: session id = c167a0a433d34b74affed13cbf96e302-f7442d12e0234db38d896c09a9e73162
I20240503 07:59:19.589951 6581 tablet_copy_source_session.cc:205] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.590665 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7442d12e0234db38d896c09a9e73162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.591122 13684 tablet_copy_client.cc:783] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.591272 13684 tablet_copy_client.cc:647] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.592267 13684 tablet_copy_client.cc:523] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.592516 13677 tablet_copy_client.cc:783] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.592674 13677 tablet_copy_client.cc:647] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.593431 13684 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:19.593649 13677 tablet_copy_client.cc:523] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.594514 13677 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:19.596848 13684 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.597105 13684 tablet_bootstrap.cc:492] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:19.597292 13677 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.597338 13684 ts_tablet_manager.cc:1367] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:19.597503 13677 tablet_bootstrap.cc:492] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Bootstrap complete.
I20240503 07:59:19.597633 13677 ts_tablet_manager.cc:1367] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:19.597959 13684 raft_consensus.cc:348] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.598059 13684 raft_consensus.cc:729] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:19.598141 13684 consensus_queue.cc:260] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.598140 13677 raft_consensus.cc:348] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.598244 13677 raft_consensus.cc:729] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: LEARNER
I20240503 07:59:19.598392 13684 ts_tablet_manager.cc:1398] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.598410 13677 consensus_queue.cc:260] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.598740 13677 ts_tablet_manager.cc:1398] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.599306 8997 tablet_copy_service.cc:342] P 3d08a06d53b14381bee6223c4e1a6027: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-e1b2ea716ca64e309efccd291db503d6 received from {username='slave'} at 127.0.99.130:56247
I20240503 07:59:19.599366 8997 tablet_copy_service.cc:433] P 3d08a06d53b14381bee6223c4e1a6027: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-e1b2ea716ca64e309efccd291db503d6 on tablet e1b2ea716ca64e309efccd291db503d6 with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:19.599522 6581 tablet_copy_service.cc:342] P ad910f92ef9d4db8a8962cecba289f8a: Request end of tablet copy session c167a0a433d34b74affed13cbf96e302-f7442d12e0234db38d896c09a9e73162 received from {username='slave'} at 127.0.99.129:58657
I20240503 07:59:19.599588 6581 tablet_copy_service.cc:433] P ad910f92ef9d4db8a8962cecba289f8a: ending tablet copy session c167a0a433d34b74affed13cbf96e302-f7442d12e0234db38d896c09a9e73162 on tablet f7442d12e0234db38d896c09a9e73162 with peer c167a0a433d34b74affed13cbf96e302
I20240503 07:59:19.605502 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:19.612051 8560 tablet_service.cc:1467] Processing CreateTablet for tablet acdb48f75a014b8d9e82f045950f4c98 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.612340 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdb48f75a014b8d9e82f045950f4c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.612404 8822 tablet_service.cc:1467] Processing CreateTablet for tablet d7483dd58cbd450090e2935306118740 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.612432 6539 tablet_service.cc:1467] Processing CreateTablet for tablet d7483dd58cbd450090e2935306118740 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.612650 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7483dd58cbd450090e2935306118740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.612695 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7483dd58cbd450090e2935306118740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.614993 13249 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:19.615105 8955 tablet_service.cc:1467] Processing CreateTablet for tablet d7483dd58cbd450090e2935306118740 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.615332 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7483dd58cbd450090e2935306118740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.615823 13249 tablet_bootstrap.cc:654] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.616480 8957 tablet_service.cc:1467] Processing CreateTablet for tablet acdb48f75a014b8d9e82f045950f4c98 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.616693 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdb48f75a014b8d9e82f045950f4c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.617811 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 30616a5a684b4cfda26ad3543f11b07c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.618053 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30616a5a684b4cfda26ad3543f11b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.618680 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 30616a5a684b4cfda26ad3543f11b07c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.618875 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30616a5a684b4cfda26ad3543f11b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.620026 6670 tablet_service.cc:1467] Processing CreateTablet for tablet acdb48f75a014b8d9e82f045950f4c98 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.620234 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdb48f75a014b8d9e82f045950f4c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.620982 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 30616a5a684b4cfda26ad3543f11b07c (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=1685b982282a432790544440ddf1116a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:19.621177 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30616a5a684b4cfda26ad3543f11b07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.624812 13708 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:19.626000 13709 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:19.627902 13709 tablet_bootstrap.cc:654] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.628006 13706 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:19.628962 13708 tablet_bootstrap.cc:654] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.629048 13706 tablet_bootstrap.cc:654] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.631134 13709 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:19.631214 13709 ts_tablet_manager.cc:1367] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:19.631596 13709 raft_consensus.cc:348] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.631644 13249 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:19.631688 13709 raft_consensus.cc:374] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.631714 13709 raft_consensus.cc:729] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.631723 13249 ts_tablet_manager.cc:1367] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.000s
I20240503 07:59:19.631831 13709 consensus_queue.cc:260] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.632073 13709 ts_tablet_manager.cc:1398] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.632107 13249 raft_consensus.cc:348] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.632195 13709 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:19.632206 13249 raft_consensus.cc:374] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.632239 13249 raft_consensus.cc:729] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.632361 13249 consensus_queue.cc:260] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.632625 13249 ts_tablet_manager.cc:1398] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.632786 13249 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:19.633028 13709 tablet_bootstrap.cc:654] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.633728 13249 tablet_bootstrap.cc:654] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.635351 13708 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:19.635491 13708 ts_tablet_manager.cc:1367] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:19.635931 13708 raft_consensus.cc:348] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.636082 13708 raft_consensus.cc:374] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.636154 13708 raft_consensus.cc:729] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.636349 13708 consensus_queue.cc:260] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.638458 13249 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:19.638530 13249 ts_tablet_manager.cc:1367] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:19.638866 13708 ts_tablet_manager.cc:1398] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:19.638875 13249 raft_consensus.cc:348] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.638952 13249 raft_consensus.cc:374] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.638976 13249 raft_consensus.cc:729] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.639022 13708 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:19.639065 13249 consensus_queue.cc:260] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.639271 13249 ts_tablet_manager.cc:1398] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.639900 13708 tablet_bootstrap.cc:654] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.640501 13706 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:19.640581 13706 ts_tablet_manager.cc:1367] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:19.641095 13706 raft_consensus.cc:348] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.641197 13706 raft_consensus.cc:374] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.641234 13706 raft_consensus.cc:729] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.641350 13706 consensus_queue.cc:260] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.641608 13706 ts_tablet_manager.cc:1398] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.642127 13708 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:19.642192 13708 ts_tablet_manager.cc:1367] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:19.642568 13708 raft_consensus.cc:348] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.642653 13708 raft_consensus.cc:374] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.642680 13708 raft_consensus.cc:729] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.642764 13708 consensus_queue.cc:260] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.642982 13708 ts_tablet_manager.cc:1398] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.643095 13708 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:19.645059 13720 ts_tablet_manager.cc:902] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:19.644850 13705 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:19.646700 13720 tablet_copy_client.cc:308] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:19.647068 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 3e889be4506e4f01a978f9e94f6b18d2 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:56623)
I20240503 07:59:19.647131 13709 tablet_bootstrap.cc:492] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:19.647153 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 3e889be4506e4f01a978f9e94f6b18d2 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:56623: session id = ad910f92ef9d4db8a8962cecba289f8a-3e889be4506e4f01a978f9e94f6b18d2
I20240503 07:59:19.647202 13709 ts_tablet_manager.cc:1367] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240503 07:59:19.647586 13709 raft_consensus.cc:348] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.647673 13709 raft_consensus.cc:374] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.647701 13709 raft_consensus.cc:729] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.647805 13709 consensus_queue.cc:260] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.647964 12608 raft_consensus.cc:482] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.648043 13709 ts_tablet_manager.cc:1398] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.648031 12608 raft_consensus.cc:504] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.648061 6712 tablet_copy_source_session.cc:205] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.648335 12608 leader_election.cc:288] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:19.649029 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
W20240503 07:59:19.649384 6496 leader_election.cc:341] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:19.650554 13705 tablet_bootstrap.cc:654] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.643821 13708 tablet_bootstrap.cc:654] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.648921 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:19.650805 8976 raft_consensus.cc:2454] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:19.651014 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e889be4506e4f01a978f9e94f6b18d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.651082 6496 leader_election.cc:302] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters: 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:19.651787 12608 raft_consensus.cc:2781] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.651844 12608 raft_consensus.cc:482] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.651876 12608 raft_consensus.cc:3037] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.652673 12608 raft_consensus.cc:504] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.653016 12608 leader_election.cc:288] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:19.653704 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:19.653016 13720 tablet_copy_client.cc:783] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.653874 13720 tablet_copy_client.cc:647] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
W20240503 07:59:19.654003 6496 leader_election.cc:341] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:19.654163 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:19.654260 8976 raft_consensus.cc:3037] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.654919 13708 tablet_bootstrap.cc:492] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:19.655040 13708 ts_tablet_manager.cc:1367] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:19.655103 8976 raft_consensus.cc:2454] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:19.655375 6496 leader_election.cc:302] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters: 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:19.655573 12608 raft_consensus.cc:2781] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.655673 12608 raft_consensus.cc:686] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:19.655707 13708 raft_consensus.cc:348] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.655783 12608 consensus_queue.cc:237] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.655907 13708 raft_consensus.cc:374] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.656189 13708 raft_consensus.cc:729] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.656376 13708 consensus_queue.cc:260] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.656937 13708 ts_tablet_manager.cc:1398] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.657222 13734 ts_tablet_manager.cc:902] T 60f643dce3eb4c669b36b0714a31651b P 9553fe3c4ccd4e91b7c176aad82fec09: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
F20240503 07:59:19.657495 13734 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.queue_time_us, tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.queue_time_us) (tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.run_cpu_time_us, tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.run_cpu_time_us) (tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.run_wall_time_us, tablet-download-pool-0e3f58a27f2a4a2593e8219b1d7bd894.run_wall_time_us) (tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.queue_time_us, tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.queue_time_us) (tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.run_cpu_time_us, tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.run_cpu_time_us) (tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.run_wall_time_us, tablet-download-pool-0e9ce52b9ae543cca72fb92efafc3c4d.run_wall_time_us) (tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.queue_time_us, tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.queue_time_us) (tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.run_cpu_time_us, tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.run_cpu_time_us) (tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.run_wall_time_us, tablet-download-pool-11fbfb84cef74316a77b3ccfab49a19b.run_wall_time_us) (tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.queue_time_us, tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.queue_time_us) (tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.run_cpu_time_us, tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.run_cpu_time_us) (tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.run_wall_time_us, tablet-download-pool-2a10085c34be4d31aae9f56a2ec1ba23.run_wall_time_us) (tablet-download-pool-435f73fb9f384b10b353df19037bcac3.queue_time_us, tablet-download-pool-435f73fb9f384b10b353df19037bcac3.queue_time_us) (tablet-download-pool-435f73fb9f384b10b353df19037bcac3.run_cpu_time_us, tablet-download-pool-435f73fb9f384b10b353df19037bcac3.run_cpu_time_us) (tablet-download-pool-435f73fb9f384b10b353df19037bcac3.run_wall_time_us, tablet-download-pool-435f73fb9f384b10b353df19037bcac3.run_wall_time_us) (tablet-download-pool-5f05300c9002408084f0b47d5e47591d.queue_time_us, tablet-download-pool-5f05300c9002408084f0b47d5e47591d.queue_time_us) (tablet-download-pool-5f05300c9002408084f0b47d5e47591d.run_cpu_time_us, tablet-download-pool-5f05300c9002408084f0b47d5e47591d.run_cpu_time_us) (tablet-download-pool-5f05300c9002408084f0b47d5e47591d.run_wall_time_us, tablet-download-pool-5f05300c9002408084f0b47d5e47591d.run_wall_time_us) (tablet-download-pool-60f643dce3eb4c669b36b0714a31651b.queue_time_us, tablet-download-pool-60f643dce3eb4c669b36b0714a31651b.queue_time_us) (tablet-download-pool-7f417c61594348428ee9514ded49ee88.queue_time_us, tablet-download-pool-7f417c61594348428ee9514ded49ee88.queue_time_us) (tablet-download-pool-7f417c61594348428ee9514ded49ee88.run_cpu_time_us, tablet-download-pool-7f417c61594348428ee9514ded49ee88.run_cpu_time_us) (tablet-download-pool-7f417c61594348428ee9514ded49ee88.run_wall_time_us, tablet-download-pool-7f417c61594348428ee9514ded49ee88.run_wall_time_us) (tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.queue_time_us, tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.queue_time_us) (tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.run_cpu_time_us, tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.run_cpu_time_us) (tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.run_wall_time_us, tablet-download-pool-8384cbf3f368424d9f8a9060efe9c0f2.run_wall_time_us) (tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.queue_time_us, tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.queue_time_us) (tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.run_cpu_time_us, tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.run_cpu_time_us) (tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.run_wall_time_us, tablet-download-pool-974c815ff8e94190a7bf28444cb3cd9f.run_wall_time_us) (tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.queue_time_us, tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.queue_time_us) (tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.run_cpu_time_us, tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.run_cpu_time_us) (tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.run_wall_time_us, tablet-download-pool-9772b487ca864ad98e9d79d8435c34c6.run_wall_time_us) (tablet-download-pool-afcd161598db41d7962caca61f63ef77.queue_time_us, tablet-download-pool-afcd161598db41d7962caca61f63ef77.queue_time_us) (tablet-download-pool-afcd161598db41d7962caca61f63ef77.run_cpu_time_us, tablet-download-pool-afcd161598db41d7962caca61f63ef77.run_cpu_time_us) (tablet-download-pool-afcd161598db41d7962caca61f63ef77.run_wall_time_us, tablet-download-pool-afcd161598db41d7962caca61f63ef77.run_wall_time_us) (tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.queue_time_us, tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.queue_time_us) (tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.run_cpu_time_us, tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.run_cpu_time_us) (tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.run_wall_time_us, tablet-download-pool-bfd4259777e5492c904e6d65a36d92ed.run_wall_time_us) (tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.queue_time_us, tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.queue_time_us) (tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.run_cpu_time_us, tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.run_cpu_time_us) (tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.run_wall_time_us, tablet-download-pool-edf2c780d0d64db79dca88eac77d4478.run_wall_time_us) (tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.queue_time_us, tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.queue_time_us) (tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.run_cpu_time_us, tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.run_cpu_time_us) (tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.run_wall_time_us, tablet-download-pool-fd67fb07203e4e99919ec0317bdf077d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 07:59:19.658118 13720 tablet_copy_client.cc:523] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
*** Check failure stack trace: ***
I20240503 07:59:19.658779 13705 tablet_bootstrap.cc:492] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:19.658704 6153 catalog_manager.cc:5484] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.658880 13705 ts_tablet_manager.cc:1367] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:19.659251 13720 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:19.659286 13705 raft_consensus.cc:348] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.659399 13705 raft_consensus.cc:374] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.659431 13705 raft_consensus.cc:729] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.659557 13705 consensus_queue.cc:260] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.659843 13705 ts_tablet_manager.cc:1398] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.662487 13720 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.662712 13720 tablet_bootstrap.cc:492] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:19.662854 13720 ts_tablet_manager.cc:1367] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 07:59:19.663689 13720 raft_consensus.cc:348] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:19.663925 13720 raft_consensus.cc:729] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:19.664042 13720 consensus_queue.cc:260] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
@ 0x7f21e306cdcd google::LogMessage::Fail() at ??:0
I20240503 07:59:19.664887 13720 ts_tablet_manager.cc:1398] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:19.665305 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-3e889be4506e4f01a978f9e94f6b18d2 received from {username='slave'} at 127.0.99.131:56623
I20240503 07:59:19.665370 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-3e889be4506e4f01a978f9e94f6b18d2 on tablet 3e889be4506e4f01a978f9e94f6b18d2 with peer ad910f92ef9d4db8a8962cecba289f8a
@ 0x7f21e3070b93 google::LogMessage::SendToLog() at ??:0
I20240503 07:59:19.668283 6153 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:19.668382 6153 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
@ 0x7f21e306c7cc google::LogMessage::Flush() at ??:0
I20240503 07:59:19.670287 6153 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 0fe511d140c8487b9ede11a841d48bba
@ 0x7f21e306df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 07:59:19.670809 8691 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe511d140c8487b9ede11a841d48bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:59772
I20240503 07:59:19.671059 13742 tablet_replica.cc:330] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: stopping tablet replica
I20240503 07:59:19.671185 13742 raft_consensus.cc:2229] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.671226 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe511d140c8487b9ede11a841d48bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:19.671284 13742 raft_consensus.cc:2258] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.671260 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe511d140c8487b9ede11a841d48bba with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:19 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:19.671523 13743 tablet_replica.cc:330] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:19.671645 13743 raft_consensus.cc:2229] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:19.671849 13743 raft_consensus.cc:2258] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.672441 13743 ts_tablet_manager.cc:1875] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:19.673009 13742 ts_tablet_manager.cc:1875] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f21e44d6044 kudu::TraceMetrics::InternName() at ??:0
I20240503 07:59:19.674619 13743 ts_tablet_manager.cc:1888] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.674679 13743 log.cc:1196] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0fe511d140c8487b9ede11a841d48bba
I20240503 07:59:19.674929 13743 ts_tablet_manager.cc:1909] T 0fe511d140c8487b9ede11a841d48bba P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
@ 0x7f21e44baf98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 07:59:19.676004 13742 ts_tablet_manager.cc:1888] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.676065 13742 log.cc:1196] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/0fe511d140c8487b9ede11a841d48bba
I20240503 07:59:19.676318 13742 ts_tablet_manager.cc:1909] T 0fe511d140c8487b9ede11a841d48bba P 9553fe3c4ccd4e91b7c176aad82fec09: Deleting consensus metadata
I20240503 07:59:19.676993 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table_new_name [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]) successfully deleted
I20240503 07:59:19.677058 6144 catalog_manager.cc:4830] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table_new_name [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]) successfully deleted
I20240503 07:59:19.677421 13684 ts_tablet_manager.cc:902] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:19.677819 13684 tablet_copy_client.cc:308] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:39417
I20240503 07:59:19.678272 8864 tablet_copy_service.cc:140] P 616fdda6ba714cb7be01a564de2b9db2: Received BeginTabletCopySession request for tablet fc7d26466129405bbe6db2721823bae1 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:36371)
I20240503 07:59:19.678350 8864 tablet_copy_service.cc:161] P 616fdda6ba714cb7be01a564de2b9db2: Beginning new tablet copy session on tablet fc7d26466129405bbe6db2721823bae1 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:36371: session id = d1f36d57661b4429a4be8d10b58d8ff7-fc7d26466129405bbe6db2721823bae1
I20240503 07:59:19.679103 8864 tablet_copy_source_session.cc:205] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.679842 13684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7d26466129405bbe6db2721823bae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.680976 13744 tablet_replica.cc:330] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:19.681121 13744 raft_consensus.cc:2229] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:19.681373 13744 raft_consensus.cc:2258] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:19.682178 13684 tablet_copy_client.cc:783] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.682310 13684 tablet_copy_client.cc:647] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
@ 0x7f21e44b8efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 07:59:19.683629 13754 ts_tablet_manager.cc:902] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Initiating tablet copy from peer 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:19.683817 13684 tablet_copy_client.cc:523] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.684346 13754 tablet_copy_client.cc:308] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:39417
I20240503 07:59:19.685556 13744 ts_tablet_manager.cc:1875] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f21e706b77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 07:59:19.684953 13684 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:19.687858 8864 tablet_copy_service.cc:140] P 616fdda6ba714cb7be01a564de2b9db2: Received BeginTabletCopySession request for tablet cfd4deb6a5104141bfa51fe4114bb708 from peer d1f36d57661b4429a4be8d10b58d8ff7 ({username='slave'} at 127.0.99.130:36371)
I20240503 07:59:19.687937 8864 tablet_copy_service.cc:161] P 616fdda6ba714cb7be01a564de2b9db2: Beginning new tablet copy session on tablet cfd4deb6a5104141bfa51fe4114bb708 from peer d1f36d57661b4429a4be8d10b58d8ff7 at {username='slave'} at 127.0.99.130:36371: session id = d1f36d57661b4429a4be8d10b58d8ff7-cfd4deb6a5104141bfa51fe4114bb708
I20240503 07:59:19.688999 8864 tablet_copy_source_session.cc:205] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.689644 13744 ts_tablet_manager.cc:1888] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:19.689700 13754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd4deb6a5104141bfa51fe4114bb708. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.689715 13744 log.cc:1196] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/0fe511d140c8487b9ede11a841d48bba
I20240503 07:59:19.690052 13744 ts_tablet_manager.cc:1909] T 0fe511d140c8487b9ede11a841d48bba P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:19.690728 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 0fe511d140c8487b9ede11a841d48bba (table rebalancer_dynamic_table_new_name [id=c1422f5b84ef4bdfb6a9e0a5ed147fea]) successfully deleted
@ 0x7f21e7074a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 07:59:19.691581 13754 tablet_copy_client.cc:783] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.691689 13754 tablet_copy_client.cc:647] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.693830 13754 tablet_copy_client.cc:523] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f21e70f576f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 07:59:19.694885 13754 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:19.698410 13684 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
@ 0x7f21e71094f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 07:59:19.698614 13684 tablet_bootstrap.cc:492] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:19.698740 13684 ts_tablet_manager.cc:1367] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240503 07:59:19.699131 13684 raft_consensus.cc:348] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.699230 13684 raft_consensus.cc:729] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:19.699321 13684 consensus_queue.cc:260] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.699398 13754 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.699591 13684 ts_tablet_manager.cc:1398] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.699589 13754 tablet_bootstrap.cc:492] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap complete.
I20240503 07:59:19.699780 13754 ts_tablet_manager.cc:1367] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:19.700062 8864 tablet_copy_service.cc:342] P 616fdda6ba714cb7be01a564de2b9db2: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-fc7d26466129405bbe6db2721823bae1 received from {username='slave'} at 127.0.99.130:36371
I20240503 07:59:19.700116 8864 tablet_copy_service.cc:433] P 616fdda6ba714cb7be01a564de2b9db2: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-fc7d26466129405bbe6db2721823bae1 on tablet fc7d26466129405bbe6db2721823bae1 with peer d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:19.700172 13754 raft_consensus.cc:348] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.700253 13754 raft_consensus.cc:729] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: LEARNER
I20240503 07:59:19.700335 13754 consensus_queue.cc:260] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: NON_VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: true } }
I20240503 07:59:19.700585 13754 ts_tablet_manager.cc:1398] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f21e70f4226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 07:59:19.704152 8864 tablet_copy_service.cc:342] P 616fdda6ba714cb7be01a564de2b9db2: Request end of tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-cfd4deb6a5104141bfa51fe4114bb708 received from {username='slave'} at 127.0.99.130:36371
I20240503 07:59:19.704211 8864 tablet_copy_service.cc:433] P 616fdda6ba714cb7be01a564de2b9db2: ending tablet copy session d1f36d57661b4429a4be8d10b58d8ff7-cfd4deb6a5104141bfa51fe4114bb708 on tablet cfd4deb6a5104141bfa51fe4114bb708 with peer d1f36d57661b4429a4be8d10b58d8ff7
@ 0x7f21e7103c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563e39de475c std::function<>::operator()() at ??:0
I20240503 07:59:19.706099 13677 ts_tablet_manager.cc:902] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Initiating tablet copy from peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:19.706467 13677 tablet_copy_client.cc:308] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:40925
I20240503 07:59:19.706856 8732 tablet_copy_service.cc:140] P 9553fe3c4ccd4e91b7c176aad82fec09: Received BeginTabletCopySession request for tablet 00bcbd5440ff4d56b1975edc4041b428 from peer c167a0a433d34b74affed13cbf96e302 ({username='slave'} at 127.0.99.129:52937)
I20240503 07:59:19.706954 8732 tablet_copy_service.cc:161] P 9553fe3c4ccd4e91b7c176aad82fec09: Beginning new tablet copy session on tablet 00bcbd5440ff4d56b1975edc4041b428 from peer c167a0a433d34b74affed13cbf96e302 at {username='slave'} at 127.0.99.129:52937: session id = c167a0a433d34b74affed13cbf96e302-00bcbd5440ff4d56b1975edc4041b428
I20240503 07:59:19.707803 8732 tablet_copy_source_session.cc:205] T 00bcbd5440ff4d56b1975edc4041b428 P 9553fe3c4ccd4e91b7c176aad82fec09: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f21e44bde06 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 07:59:19.708534 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00bcbd5440ff4d56b1975edc4041b428. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.710858 13677 tablet_copy_client.cc:783] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.710988 13677 tablet_copy_client.cc:647] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 1 WAL segments...
@ 0x7f21e44be6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 07:59:19.712832 13677 tablet_copy_client.cc:523] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.713909 13677 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
@ 0x7f21e44c003b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x563e39de475c std::function<>::operator()() at ??:0
@ 0x7f21e44ad807 kudu::Thread::SuperviseThread() at ??:0
I20240503 07:59:19.718775 13677 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.718938 13677 tablet_bootstrap.cc:492] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Bootstrap complete.
I20240503 07:59:19.719036 13677 ts_tablet_manager.cc:1367] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:19.719368 13677 raft_consensus.cc:348] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.719468 13677 raft_consensus.cc:729] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: LEARNER
I20240503 07:59:19.719559 13677 consensus_queue.cc:260] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.719874 13677 ts_tablet_manager.cc:1398] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
@ 0x7f21e4ceb6db start_thread at ??:0
I20240503 07:59:19.720698 8732 tablet_copy_service.cc:342] P 9553fe3c4ccd4e91b7c176aad82fec09: Request end of tablet copy session c167a0a433d34b74affed13cbf96e302-00bcbd5440ff4d56b1975edc4041b428 received from {username='slave'} at 127.0.99.129:52937
I20240503 07:59:19.720760 8732 tablet_copy_service.cc:433] P 9553fe3c4ccd4e91b7c176aad82fec09: ending tablet copy session c167a0a433d34b74affed13cbf96e302-00bcbd5440ff4d56b1975edc4041b428 on tablet 00bcbd5440ff4d56b1975edc4041b428 with peer c167a0a433d34b74affed13cbf96e302
I20240503 07:59:19.721457 13478 raft_consensus.cc:482] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.721537 13478 raft_consensus.cc:504] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:19.721889 13478 leader_election.cc:288] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
@ 0x7f21e222371f clone at ??:0
I20240503 07:59:19.722168 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:19.722914 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7483dd58cbd450090e2935306118740" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:19.723053 8976 raft_consensus.cc:2379] T d7483dd58cbd450090e2935306118740 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 616fdda6ba714cb7be01a564de2b9db2 in current term 1: Already voted for candidate ad910f92ef9d4db8a8962cecba289f8a in this term.
I20240503 07:59:19.723558 8778 leader_election.cc:302] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2; no voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:19.723771 13478 raft_consensus.cc:3037] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.724706 13478 raft_consensus.cc:2726] T d7483dd58cbd450090e2935306118740 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:19.751084 13677 ts_tablet_manager.cc:902] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:19.751492 13677 tablet_copy_client.cc:308] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:19.751982 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet 8c01e3ce57874e24ae75a41243fdeaaf from peer c167a0a433d34b74affed13cbf96e302 ({username='slave'} at 127.0.99.129:55321)
I20240503 07:59:19.752089 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet 8c01e3ce57874e24ae75a41243fdeaaf from peer c167a0a433d34b74affed13cbf96e302 at {username='slave'} at 127.0.99.129:55321: session id = c167a0a433d34b74affed13cbf96e302-8c01e3ce57874e24ae75a41243fdeaaf
I20240503 07:59:19.753119 6450 tablet_copy_source_session.cc:205] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.753993 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c01e3ce57874e24ae75a41243fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.756139 13677 tablet_copy_client.cc:783] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.756283 13677 tablet_copy_client.cc:647] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.757777 13677 tablet_copy_client.cc:523] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.758972 13677 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:19.768028 13278 raft_consensus.cc:482] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.768117 13278 raft_consensus.cc:504] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.768483 13278 leader_election.cc:288] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.768874 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdb48f75a014b8d9e82f045950f4c98" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:19.768997 8976 raft_consensus.cc:2454] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:19.769129 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdb48f75a014b8d9e82f045950f4c98" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:19.769249 8575 raft_consensus.cc:2454] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:19.769546 6627 leader_election.cc:302] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:19.769907 13278 raft_consensus.cc:2781] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.769958 13278 raft_consensus.cc:482] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.769990 13278 raft_consensus.cc:3037] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.770903 13278 raft_consensus.cc:504] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.771236 13278 leader_election.cc:288] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.771497 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdb48f75a014b8d9e82f045950f4c98" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:19.771600 8976 raft_consensus.cc:3037] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.772485 8976 raft_consensus.cc:2454] T acdb48f75a014b8d9e82f045950f4c98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:19.772517 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:19.774623 6627 leader_election.cc:302] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:19.774948 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdb48f75a014b8d9e82f045950f4c98" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:19.775048 8575 raft_consensus.cc:3037] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.775974 8575 raft_consensus.cc:2454] T acdb48f75a014b8d9e82f045950f4c98 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:19.777128 13669 ts_tablet_manager.cc:902] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:19.777599 13669 tablet_copy_client.cc:308] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:37267
I20240503 07:59:19.778007 6318 tablet_copy_service.cc:140] P c167a0a433d34b74affed13cbf96e302: Received BeginTabletCopySession request for tablet 551f25dcee034b7894498e3c748b0f0e from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:50501)
I20240503 07:59:19.778093 6318 tablet_copy_service.cc:161] P c167a0a433d34b74affed13cbf96e302: Beginning new tablet copy session on tablet 551f25dcee034b7894498e3c748b0f0e from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:50501: session id = a97073f89fd44702bccb74a9b76f858d-551f25dcee034b7894498e3c748b0f0e
I20240503 07:59:19.778496 12930 raft_consensus.cc:482] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.778571 12930 raft_consensus.cc:504] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.778913 12930 leader_election.cc:288] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:19.779002 6318 tablet_copy_source_session.cc:205] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.779631 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30616a5a684b4cfda26ad3543f11b07c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:19.779736 6690 raft_consensus.cc:2454] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:19.779851 13669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551f25dcee034b7894498e3c748b0f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.780707 8912 leader_election.cc:302] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:19.781546 12930 raft_consensus.cc:2781] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.781601 12930 raft_consensus.cc:482] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.781637 12930 raft_consensus.cc:3037] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.782121 13669 tablet_copy_client.cc:783] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.782253 13669 tablet_copy_client.cc:647] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.782636 12930 raft_consensus.cc:504] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.783039 12930 leader_election.cc:288] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:19.783514 8579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30616a5a684b4cfda26ad3543f11b07c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:19.783622 8579 raft_consensus.cc:3037] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.783813 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30616a5a684b4cfda26ad3543f11b07c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:19.783917 6690 raft_consensus.cc:3037] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.784547 8579 raft_consensus.cc:2454] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:19.784633 13669 tablet_copy_client.cc:523] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.784857 6690 raft_consensus.cc:2454] T 30616a5a684b4cfda26ad3543f11b07c P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:19.785267 8910 leader_election.cc:302] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.785863 13669 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:19.786794 9347 raft_consensus.cc:2781] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.786880 9347 raft_consensus.cc:686] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:19.786998 9347 consensus_queue.cc:237] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.790290 6151 catalog_manager.cc:5484] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:19.790555 11524 raft_consensus.cc:2781] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.790637 11524 raft_consensus.cc:686] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:19.790756 11524 consensus_queue.cc:237] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:19.791400 8822 tablet_service.cc:1467] Processing CreateTablet for tablet dbd4400894d14f839670d209468f1929 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47]), partition=
I20240503 07:59:19.791671 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd4400894d14f839670d209468f1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.793447 13669 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.793668 13669 tablet_bootstrap.cc:492] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:19.793795 13669 ts_tablet_manager.cc:1367] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 07:59:19.794111 8560 tablet_service.cc:1467] Processing CreateTablet for tablet dbd4400894d14f839670d209468f1929 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47]), partition=
I20240503 07:59:19.794166 13705 tablet_bootstrap.cc:492] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:19.794328 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd4400894d14f839670d209468f1929. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.794401 13669 raft_consensus.cc:348] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.794508 13669 raft_consensus.cc:729] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:19.794589 13669 consensus_queue.cc:260] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.794831 13669 ts_tablet_manager.cc:1398] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.794992 13705 tablet_bootstrap.cc:654] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.795214 6318 tablet_copy_service.cc:342] P c167a0a433d34b74affed13cbf96e302: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-551f25dcee034b7894498e3c748b0f0e received from {username='slave'} at 127.0.99.132:50501
I20240503 07:59:19.795272 6318 tablet_copy_service.cc:433] P c167a0a433d34b74affed13cbf96e302: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-551f25dcee034b7894498e3c748b0f0e on tablet 551f25dcee034b7894498e3c748b0f0e with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:19.796005 13705 tablet_bootstrap.cc:492] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:19.796079 13705 ts_tablet_manager.cc:1367] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:19.796447 13249 tablet_bootstrap.cc:492] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:19.796479 13705 raft_consensus.cc:348] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.796568 13705 raft_consensus.cc:374] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.796594 13705 raft_consensus.cc:729] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.796700 13705 consensus_queue.cc:260] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.796990 13705 ts_tablet_manager.cc:1398] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:19.797443 13249 tablet_bootstrap.cc:654] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:19.798643 13249 tablet_bootstrap.cc:492] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:19.798717 13249 ts_tablet_manager.cc:1367] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:19.799121 13249 raft_consensus.cc:348] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.799207 13249 raft_consensus.cc:374] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:19.799238 13249 raft_consensus.cc:729] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:19.799348 13249 consensus_queue.cc:260] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.799620 13249 ts_tablet_manager.cc:1398] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.799631 13677 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.800964 13677 tablet_bootstrap.cc:492] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Bootstrap complete.
I20240503 07:59:19.801120 13677 ts_tablet_manager.cc:1367] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.000s
I20240503 07:59:19.801515 13677 raft_consensus.cc:348] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.801620 13677 raft_consensus.cc:729] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: LEARNER
I20240503 07:59:19.801702 13677 consensus_queue.cc:260] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:19.801972 13677 ts_tablet_manager.cc:1398] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:19.803907 13824 ts_tablet_manager.cc:902] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.804279 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session c167a0a433d34b74affed13cbf96e302-8c01e3ce57874e24ae75a41243fdeaaf received from {username='slave'} at 127.0.99.129:55321
I20240503 07:59:19.804356 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session c167a0a433d34b74affed13cbf96e302-8c01e3ce57874e24ae75a41243fdeaaf on tablet 8c01e3ce57874e24ae75a41243fdeaaf with peer c167a0a433d34b74affed13cbf96e302
I20240503 07:59:19.805415 13824 tablet_copy_client.cc:308] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:39949
I20240503 07:59:19.805837 8601 tablet_copy_service.cc:140] P c8d71787690245e681c85a087086e7ed: Received BeginTabletCopySession request for tablet fa8b855a6985404da6bd34daa0eb9ae7 from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:32773)
I20240503 07:59:19.805930 8601 tablet_copy_service.cc:161] P c8d71787690245e681c85a087086e7ed: Beginning new tablet copy session on tablet fa8b855a6985404da6bd34daa0eb9ae7 from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:32773: session id = a97073f89fd44702bccb74a9b76f858d-fa8b855a6985404da6bd34daa0eb9ae7
I20240503 07:59:19.806762 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30616a5a684b4cfda26ad3543f11b07c" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:19.806846 8601 tablet_copy_source_session.cc:205] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.806877 8575 raft_consensus.cc:2362] T 30616a5a684b4cfda26ad3543f11b07c P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1. Re-sending same reply.
I20240503 07:59:19.807847 13824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8b855a6985404da6bd34daa0eb9ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.807790 6151 catalog_manager.cc:5484] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:19.809986 13824 tablet_copy_client.cc:783] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.810158 13824 tablet_copy_client.cc:647] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.811230 13824 tablet_copy_client.cc:523] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.812288 13824 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:19.815372 13824 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.815593 13824 tablet_bootstrap.cc:492] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:19.815735 13824 ts_tablet_manager.cc:1367] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.001s
I20240503 07:59:19.816107 13824 raft_consensus.cc:348] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.816206 13824 raft_consensus.cc:729] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:19.816335 13824 consensus_queue.cc:260] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:19.816639 13824 ts_tablet_manager.cc:1398] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:19.817106 8601 tablet_copy_service.cc:342] P c8d71787690245e681c85a087086e7ed: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-fa8b855a6985404da6bd34daa0eb9ae7 received from {username='slave'} at 127.0.99.132:32773
I20240503 07:59:19.817188 8601 tablet_copy_service.cc:433] P c8d71787690245e681c85a087086e7ed: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-fa8b855a6985404da6bd34daa0eb9ae7 on tablet fa8b855a6985404da6bd34daa0eb9ae7 with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:19.826751 13720 ts_tablet_manager.cc:902] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Initiating tablet copy from peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:19.827216 13720 tablet_copy_client.cc:308] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:39733
I20240503 07:59:19.827666 6712 tablet_copy_service.cc:140] P a97073f89fd44702bccb74a9b76f858d: Received BeginTabletCopySession request for tablet 2054e7790d7745c2a1b08e903c64d1b7 from peer ad910f92ef9d4db8a8962cecba289f8a ({username='slave'} at 127.0.99.131:56623)
I20240503 07:59:19.827759 6712 tablet_copy_service.cc:161] P a97073f89fd44702bccb74a9b76f858d: Beginning new tablet copy session on tablet 2054e7790d7745c2a1b08e903c64d1b7 from peer ad910f92ef9d4db8a8962cecba289f8a at {username='slave'} at 127.0.99.131:56623: session id = ad910f92ef9d4db8a8962cecba289f8a-2054e7790d7745c2a1b08e903c64d1b7
I20240503 07:59:19.828573 6712 tablet_copy_source_session.cc:205] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:19.829368 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2054e7790d7745c2a1b08e903c64d1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:19.831073 13720 tablet_copy_client.cc:783] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:19.831210 13720 tablet_copy_client.cc:647] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:19.832580 13720 tablet_copy_client.cc:523] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:19.833822 13720 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:19.836521 13720 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:19.836793 13720 tablet_bootstrap.cc:492] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap complete.
I20240503 07:59:19.836952 13720 ts_tablet_manager.cc:1367] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:19.837306 13720 raft_consensus.cc:348] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:19.837427 13720 raft_consensus.cc:729] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: LEARNER
I20240503 07:59:19.837536 13720 consensus_queue.cc:260] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: NON_VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: true } }
I20240503 07:59:19.837774 13720 ts_tablet_manager.cc:1398] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:19.839969 6712 tablet_copy_service.cc:342] P a97073f89fd44702bccb74a9b76f858d: Request end of tablet copy session ad910f92ef9d4db8a8962cecba289f8a-2054e7790d7745c2a1b08e903c64d1b7 received from {username='slave'} at 127.0.99.131:56623
I20240503 07:59:19.840032 6712 tablet_copy_service.cc:433] P a97073f89fd44702bccb74a9b76f858d: ending tablet copy session ad910f92ef9d4db8a8962cecba289f8a-2054e7790d7745c2a1b08e903c64d1b7 on tablet 2054e7790d7745c2a1b08e903c64d1b7 with peer ad910f92ef9d4db8a8962cecba289f8a
I20240503 07:59:19.853379 13478 raft_consensus.cc:482] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:19.853492 13478 raft_consensus.cc:504] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.853914 13478 leader_election.cc:288] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.854246 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd4400894d14f839670d209468f1929" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:19.854413 8575 raft_consensus.cc:2454] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:19.854650 8776 leader_election.cc:302] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.854827 13478 raft_consensus.cc:2781] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:19.854898 13478 raft_consensus.cc:482] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:19.854944 13478 raft_consensus.cc:3037] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.855844 13478 raft_consensus.cc:504] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.856195 13478 leader_election.cc:288] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:19.856540 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd4400894d14f839670d209468f1929" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:19.856637 8575 raft_consensus.cc:3037] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:19.857532 8575 raft_consensus.cc:2454] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:19.857784 8776 leader_election.cc:302] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:19.858028 13478 raft_consensus.cc:2781] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:19.858108 13478 raft_consensus.cc:686] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:19.858224 13478 consensus_queue.cc:237] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:19.859921 6150 catalog_manager.cc:5484] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:19.899747 6690 raft_consensus.cc:1203] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 07:59:19.932102 8913 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.932269 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.932813 6235 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.932962 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.932968 8779 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933092 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933313 6366 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933320 6144 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933437 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933439 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933846 6628 connection.cc:535] client connection to 127.0.99.134:40925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.933954 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:19.934265 13837 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:19.932964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:40925 (local address 127.0.99.136:56363)
0503 07:59:19.933629 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:19.933707 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":57,"threads_started":1}
W20240503 07:59:19.934383 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.934954 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.935010 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.935043 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.935791 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): CreateTablet RPC failed for tablet dbd4400894d14f839670d209468f1929: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:19.936040 8779 leader_election.cc:334] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:19.936195 8779 leader_election.cc:334] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:19.936216 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.936292 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.936297 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.936327 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:19.936525 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.965428 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:19.969000 8575 raft_consensus.cc:1261] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:19.969506 12241 consensus_queue.cc:1035] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:19.969611 8779 consensus_peers.cc:485] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:19.972011 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47]). Expected version 1 got 0
I20240503 07:59:19.973441 13381 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:19.973737 13381 tablet.cc:1620] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:19.974699 6150 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47]). Expected version 1 got 0
I20240503 07:59:19.974989 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47]). Expected version 1 got 0
I20240503 07:59:19.974498 13872 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 07:59:19.975636 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:19.976116 13872 tablet.cc:1620] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:19.979385 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47] alter complete (version 1)
W20240503 07:59:19.979825 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:19.992924 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.007861 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.013000 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:20.013736 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.019672 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.036669 6298 raft_consensus.cc:1203] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.069260 13018 consensus_queue.cc:1035] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.075029 8956 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 07:59:20.075958 6560 raft_consensus.cc:1203] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.076373 12608 consensus_queue.cc:1035] T d7483dd58cbd450090e2935306118740 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.078547 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.082326 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:20.085088 6670 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 07:59:20.087222 6539 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 07:59:20.087533 6277 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 07:59:20.087666 8822 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 07:59:20.088380 6298 raft_consensus.cc:1203] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.091459 8560 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 07:59:20.095257 13872 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:20.095582 13872 tablet.cc:1620] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:20.095857 13381 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:20.096127 13381 tablet.cc:1620] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:20.098984 6408 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 07:59:20.110878 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a3f9b29ab4d44204a1b074d6db58cc47] alter complete (version 2)
W20240503 07:59:20.119805 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.121134 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.125470 6430 raft_consensus.cc:1203] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 07:59:20.134069 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.139125 6430 raft_consensus.cc:1203] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 07:59:20.142051 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.147895 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:20.151154 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240503 07:59:09.721897 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:09.722083 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:09.725208 9030 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:10.160985 9030 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 07:59:10.553947 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server c167a0a433d34b74affed13cbf96e302
W20240503 07:59:10.554349 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:10.554427 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554497 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.554554 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554594 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.554672 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.554788 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc23875e4c9c4380bd7862acd33adc2b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.554834 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 3d08a06d53b14381bee6223c4e1a6027 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:10.554929 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.554967 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555027 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555102 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6d629539a2874def9c039beeee9b9490: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555156 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555212 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555253 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555294 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555336 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555378 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555433 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555476 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555518 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.555558 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555598 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555640 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555706 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.555742 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.555776 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6d629539a2874def9c039beeee9b9490: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.555804 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
I20240503 07:59:10.572374 9030 rebalancer_tool.cc:1074] tablet 0e9ce52b9ae543cca72fb92efafc3c4d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.585142 9030 rebalancer_tool.cc:1074] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.615864 9030 rebalancer_tool.cc:1074] tablet c597bc1a19274f55b9d60558db35fcae: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.637740 9030 rebalancer_tool.cc:1074] tablet 610160f6ba94483a9bdcf966c413e5d1: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.656585 9030 rebalancer_tool.cc:1074] tablet e874c9cd82864973a92fb743e45ccf67: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.671169 9030 rebalancer_tool.cc:1074] tablet 5538a709d8d440cb8f006a163f259389: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.685398 9030 rebalancer_tool.cc:1074] tablet c7e3555fb9214a40bd8dbf937b87cbaa: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.699667 9030 rebalancer_tool.cc:1074] tablet 065d5d0851e840939802896f9bddb84c: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.701303 9030 rebalancer_tool.cc:1084] tablet bc0992eb4c7a471f9646518dd20e7c26: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move ignored: Not found: bc0992eb4c7a471f9646518dd20e7c26: tablet not found
I20240503 07:59:10.701377 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:10.877220 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877312 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877343 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877378 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 3d08a06d53b14381bee6223c4e1a6027 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:10.877506 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cc23875e4c9c4380bd7862acd33adc2b: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877537 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877590 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877629 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877681 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877712 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e85dfa9534124e9daa2fee791efcba8a: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server c167a0a433d34b74affed13cbf96e302
W20240503 07:59:10.877741 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.877781 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877812 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877841 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.877873 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.877902 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.877931 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.877985 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.878016 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.878062 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878091 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:10.878120 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878152 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:10.878192 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:10.878221 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:10.878253 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:10.891927 9030 rebalancer_tool.cc:1074] tablet 09c32a1e047844a3b09f2c790d735c9b: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.905635 9030 rebalancer_tool.cc:1074] tablet 8384cbf3f368424d9f8a9060efe9c0f2: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:10.918112 9030 rebalancer_tool.cc:1074] tablet 93eb2db48ef449a486a0347a76a8a5ea: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:10.931584 9030 rebalancer_tool.cc:1074] tablet 82de209205b342cfaac10506fdac9303: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:10.958356 9030 rebalancer_tool.cc:1074] tablet 5710dff79184444797bb46d981ae6f55: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:10.974042 9030 rebalancer_tool.cc:1074] tablet 7f417c61594348428ee9514ded49ee88: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.000555 9030 rebalancer_tool.cc:1074] tablet 2f943ecfe59f432c861a54b16c62572f: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:11.057148 9030 rebalancer_tool.cc:1074] tablet 5d95cc5335534f02aa62b00fb8b9c33c: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:11.093199 9030 rebalancer_tool.cc:1074] tablet 39b01318dc1543cba2cc991e9db91c28: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:11.115651 9030 rebalancer_tool.cc:1074] tablet fd67fb07203e4e99919ec0317bdf077d: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.137940 9030 rebalancer_tool.cc:1074] tablet 5631f3e97fe04013a2a7140f44260003: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:11.158378 9030 rebalancer_tool.cc:1074] tablet d4308bc87be24f28b6566ce76cae6e50: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:11.189060 9030 rebalancer_tool.cc:1123] tablet e874c9cd82864973a92fb743e45ccf67: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: e874c9cd82864973a92fb743e45ccf67: tablet not found
I20240503 07:59:11.303802 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:11.577474 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.577666 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:11.577759 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.577996 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578090 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:11.578140 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578197 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578246 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578308 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578356 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578405 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578449 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.578495 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578562 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.578608 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578685 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578749 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.578788 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:11.578874 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.578919 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.578963 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:11.579027 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.579088 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7441f4d1d4d2427288686c47216a9c95: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:11.579138 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:11.579186 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:11.579780 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:11.625955 9030 rebalancer_tool.cc:1074] tablet 5f05300c9002408084f0b47d5e47591d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:11.708299 9030 rebalancer_tool.cc:1123] tablet fd67fb07203e4e99919ec0317bdf077d: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: fd67fb07203e4e99919ec0317bdf077d: tablet not found
I20240503 07:59:11.735075 9030 rebalancer_tool.cc:1123] tablet 5710dff79184444797bb46d981ae6f55: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: 5710dff79184444797bb46d981ae6f55: tablet not found
I20240503 07:59:11.759567 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:12.220618 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220737 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220836 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.220952 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:12.220991 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221032 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4c4628ca9d9a4e16a0d6d2c1a88adccc: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221069 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221123 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 763a085c93d34fdbb045682646bf8b13: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221297 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221478 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221534 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221599 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221642 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.221684 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221741 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1a03624a2d41492c98d74f5c453afd47: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:12.221783 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.221894 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.221966 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.222004 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:12.222044 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e126a94a1cac419b895e54036cecb857: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:12.222084 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28fc17ad519645069eb8dffa88723735: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:12.222126 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 936b932ca2ae410baea29e6202e71429: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:12.239388 9030 rebalancer_tool.cc:1074] tablet 0e3f58a27f2a4a2593e8219b1d7bd894: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.257222 9030 rebalancer_tool.cc:1074] tablet 7d855e5edf05471983dfdbd40f37560f: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:12.295229 9030 rebalancer_tool.cc:1136] tablet 5538a709d8d440cb8f006a163f259389: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.304636 9030 rebalancer_tool.cc:1136] tablet 0e9ce52b9ae543cca72fb92efafc3c4d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.312577 9030 rebalancer_tool.cc:1136] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.394002 9030 rebalancer_tool.cc:1136] tablet 065d5d0851e840939802896f9bddb84c: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.398692 9030 rebalancer_tool.cc:1136] tablet 610160f6ba94483a9bdcf966c413e5d1: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.403508 9030 rebalancer_tool.cc:1136] tablet 8384cbf3f368424d9f8a9060efe9c0f2: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.413825 9030 rebalancer_tool.cc:1123] tablet 09c32a1e047844a3b09f2c790d735c9b: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: 09c32a1e047844a3b09f2c790d735c9b: tablet not found
I20240503 07:59:12.428738 9030 rebalancer_tool.cc:1136] tablet c597bc1a19274f55b9d60558db35fcae: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:12.446739 9030 rebalancer_tool.cc:1074] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.459218 9030 rebalancer_tool.cc:1074] tablet f5604af852814da58385f072eb9600aa: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.472692 9030 rebalancer_tool.cc:1074] tablet 11fbfb84cef74316a77b3ccfab49a19b: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.488082 9030 rebalancer_tool.cc:1074] tablet 9772b487ca864ad98e9d79d8435c34c6: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:12.507879 9030 rebalancer_tool.cc:1074] tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.520573 9030 rebalancer_tool.cc:1074] tablet 435f73fb9f384b10b353df19037bcac3: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.532786 9030 rebalancer_tool.cc:1074] tablet edf2c780d0d64db79dca88eac77d4478: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:12.686764 9030 rebalancer_tool.cc:1136] tablet 39b01318dc1543cba2cc991e9db91c28: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:12.692605 9030 rebalancer_tool.cc:1136] tablet 93eb2db48ef449a486a0347a76a8a5ea: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.714674 9030 rebalancer_tool.cc:1136] tablet 5631f3e97fe04013a2a7140f44260003: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.747504 9030 rebalancer_tool.cc:1136] tablet 82de209205b342cfaac10506fdac9303: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.782550 9030 rebalancer_tool.cc:1136] tablet 7f417c61594348428ee9514ded49ee88: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:12.789462 9030 rebalancer_tool.cc:1136] tablet 2f943ecfe59f432c861a54b16c62572f: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:12.798152 9030 rebalancer_tool.cc:1136] tablet 5d95cc5335534f02aa62b00fb8b9c33c: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:12.810915 9030 rebalancer_tool.cc:1074] tablet a2762b2e6de842bb8c898aea5b3910ba: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:12.828131 9030 rebalancer_tool.cc:1074] tablet 824c42be8152494fad2cbc76ece56981: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:12.829293 9030 rebalancer_tool.cc:1084] tablet dc995af1acb9459f9096867bc3b075f4: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move ignored: Not found: dc995af1acb9459f9096867bc3b075f4: tablet not found
I20240503 07:59:12.829353 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:13.115881 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.115970 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116029 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.116092 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116153 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116227 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 280837395f114a969bd89f792265a5e2: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116353 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.116434 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:13.116550 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.116612 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.116659 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:13.116703 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.116757 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.116917 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:13.145793 9030 rebalancer_tool.cc:1074] tablet b29dafc7b564493a8fdf064e7d3461ef: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.167183 9030 rebalancer_tool.cc:1074] tablet 8861b52c306e43d8b0615202cf7d8e35: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:13.195382 9030 rebalancer_tool.cc:1074] tablet a01fa99be75f498a8394aabbfabb3f5a: '9553fe3c4ccd4e91b7c176aad82fec09' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:13.211068 9030 rebalancer_tool.cc:1074] tablet 2f943ecfe59f432c861a54b16c62572f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.211911 9030 rebalancer_tool.cc:1084] tablet 18677aa766454b8c98a1a2f4a48feee2: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move ignored: Not found: 18677aa766454b8c98a1a2f4a48feee2: tablet not found
I20240503 07:59:13.211973 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:13.426029 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1b729dc47d2647d7b16c8da03ad3310c: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.426100 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 053079d7b58a495aa82a1046fa69c514: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.426265 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.426383 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1558bad4bc7b4dd99890cbbd33e36d27: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426430 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.426486 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfcfc693f3d8455f9118e86b6a99ee80: could not find any suitable replica to move from server 616fdda6ba714cb7be01a564de2b9db2 to server d1f36d57661b4429a4be8d10b58d8ff7
W20240503 07:59:13.426545 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:13.426663 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 18098099334d40cead69112e658daaef: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426784 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2040f6f2be794bbea4b293e02e8ee6e1: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server c8d71787690245e681c85a087086e7ed
W20240503 07:59:13.426831 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e90fb89304104618a7cac10234d099b3: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426872 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda5136a1c36474ca09f041825ad0c43: could not find any suitable replica to move from server a97073f89fd44702bccb74a9b76f858d to server 616fdda6ba714cb7be01a564de2b9db2
W20240503 07:59:13.426940 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 638f2ffce1de4ac6b2869df3c8574ef6: could not find any suitable replica to move from server c167a0a433d34b74affed13cbf96e302 to server 3d08a06d53b14381bee6223c4e1a6027
W20240503 07:59:13.426985 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:13.443693 9030 rebalancer_tool.cc:1074] tablet 94d5216ff8654561b44a0cfd0f9b1625: 'c167a0a433d34b74affed13cbf96e302' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:13.460263 9030 rebalancer_tool.cc:1074] tablet f534eefb7c334889a9c5d02175ae2d94: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:13.489634 9030 rebalancer_tool.cc:1123] tablet b29dafc7b564493a8fdf064e7d3461ef: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move is abandoned: Not found: b29dafc7b564493a8fdf064e7d3461ef: tablet not found
I20240503 07:59:13.550467 9030 rebalancer_tool.cc:1123] tablet 5f05300c9002408084f0b47d5e47591d: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 5f05300c9002408084f0b47d5e47591d: tablet not found
I20240503 07:59:13.556649 9030 rebalancer_tool.cc:1136] tablet d4308bc87be24f28b6566ce76cae6e50: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:13.557471 9030 rebalancer_tool.cc:1123] tablet 11fbfb84cef74316a77b3ccfab49a19b: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 11fbfb84cef74316a77b3ccfab49a19b: tablet not found
I20240503 07:59:13.609514 9030 rebalancer_tool.cc:1074] tablet b2e0f5369e8349eba7188b39ae6bac30: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:13.735255 9030 rebalancer_tool.cc:1136] tablet c7e3555fb9214a40bd8dbf937b87cbaa: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:13.736071 9030 rebalancer_tool.cc:1123] tablet 0e3f58a27f2a4a2593e8219b1d7bd894: 'c167a0a433d34b74affed13cbf96e302' -> '9553fe3c4ccd4e91b7c176aad82fec09' move is abandoned: Not found: 0e3f58a27f2a4a2593e8219b1d7bd894: tablet not found
I20240503 07:59:13.757048 9030 rebalancer_tool.cc:1136] tablet 7d855e5edf05471983dfdbd40f37560f: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:13.801738 9030 rebalancer_tool.cc:1074] tablet 5631f3e97fe04013a2a7140f44260003: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.825026 9030 rebalancer_tool.cc:1074] tablet 60f643dce3eb4c669b36b0714a31651b: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:13.888356 9030 rebalancer_tool.cc:1136] tablet f5604af852814da58385f072eb9600aa: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:13.935043 9030 rebalancer_tool.cc:1074] tablet 39b01318dc1543cba2cc991e9db91c28: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:14.044137 9030 rebalancer_tool.cc:1136] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'a97073f89fd44702bccb74a9b76f858d' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.074894 9030 rebalancer_tool.cc:1136] tablet edf2c780d0d64db79dca88eac77d4478: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:14.080552 9030 rebalancer_tool.cc:1136] tablet 9772b487ca864ad98e9d79d8435c34c6: 'c167a0a433d34b74affed13cbf96e302' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:14.102175 9030 rebalancer_tool.cc:1074] tablet bfd4259777e5492c904e6d65a36d92ed: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:14.122247 9030 rebalancer_tool.cc:1074] tablet f635e30fb6974fa1a79d8a46eecc3d5e: '3d08a06d53b14381bee6223c4e1a6027' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:14.215744 9030 rebalancer_tool.cc:1136] tablet 435f73fb9f384b10b353df19037bcac3: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:14.226025 9030 rebalancer_tool.cc:1136] tablet 4b9870eac6ad47d4a9ad5694dcaa1b1f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.264636 9030 rebalancer_tool.cc:1074] tablet 2d3f03610866454595f4b6b87b7cd9cb: 'c167a0a433d34b74affed13cbf96e302' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:14.280819 9030 rebalancer_tool.cc:1074] tablet 5060df68b95e48a8a849e77ba926e608: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:14.397436 9030 rebalancer_tool.cc:1136] tablet 824c42be8152494fad2cbc76ece56981: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:14.456878 9030 rebalancer_tool.cc:1074] tablet f4e4ca6638764fb49f3afcab7688e2b3: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:14.791864 9030 rebalancer_tool.cc:1123] tablet b2e0f5369e8349eba7188b39ae6bac30: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '616fdda6ba714cb7be01a564de2b9db2' move is abandoned: Not found: b2e0f5369e8349eba7188b39ae6bac30: tablet not found
I20240503 07:59:14.797763 9030 rebalancer_tool.cc:1136] tablet 2f943ecfe59f432c861a54b16c62572f: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:14.804311 9030 rebalancer_tool.cc:1136] tablet 8861b52c306e43d8b0615202cf7d8e35: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.844938 9030 rebalancer_tool.cc:1136] tablet a2762b2e6de842bb8c898aea5b3910ba: 'ad910f92ef9d4db8a8962cecba289f8a' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:14.871966 9030 rebalancer_tool.cc:1074] tablet 558c0eb4a4334dbe91ac60fad4e820dc: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move scheduled
I20240503 07:59:14.874138 9030 rebalancer_tool.cc:1084] tablet aa754582dc9e4518965f3a2c1836cbee: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move ignored: Not found: aa754582dc9e4518965f3a2c1836cbee: tablet not found
I20240503 07:59:14.874200 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 07:59:15.148578 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5d3d4c5df5243378c8176c155dc3c5b: could not find any suitable replica to move from server ad910f92ef9d4db8a8962cecba289f8a to server 9553fe3c4ccd4e91b7c176aad82fec09
W20240503 07:59:15.148819 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 751d986bca094ff3b81b8f94513d43ec: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:15.148880 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bc13ce7b7a704afbb59e119a72b305cd: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server a97073f89fd44702bccb74a9b76f858d
W20240503 07:59:15.148983 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table af669f53d59f4eb4bdc4b498325400dd: could not find any suitable replica to move from server d1f36d57661b4429a4be8d10b58d8ff7 to server 3d08a06d53b14381bee6223c4e1a6027
I20240503 07:59:15.170928 9030 rebalancer_tool.cc:1074] tablet c5b4c776eac24f739d85e09707cd751b: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:15.191627 9030 rebalancer_tool.cc:1074] tablet c8cf3d6a3ace46769854ed61f340fb25: '616fdda6ba714cb7be01a564de2b9db2' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:15.207793 9030 rebalancer_tool.cc:1074] tablet 551f25dcee034b7894498e3c748b0f0e: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:15.220139 9030 rebalancer_tool.cc:1074] tablet 974c815ff8e94190a7bf28444cb3cd9f: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:15.240937 9030 rebalancer_tool.cc:1074] tablet e103e0e9cb16416dba7f8bbfc08ceac8: '3d08a06d53b14381bee6223c4e1a6027' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:15.262666 9030 rebalancer_tool.cc:1074] tablet 82de209205b342cfaac10506fdac9303: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:15.355584 9030 rebalancer_tool.cc:1136] tablet f534eefb7c334889a9c5d02175ae2d94: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:15.359935 9030 rebalancer_tool.cc:1136] tablet a01fa99be75f498a8394aabbfabb3f5a: '9553fe3c4ccd4e91b7c176aad82fec09' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.364805 9030 rebalancer_tool.cc:1136] tablet 94d5216ff8654561b44a0cfd0f9b1625: 'c167a0a433d34b74affed13cbf96e302' -> 'a97073f89fd44702bccb74a9b76f858d' move completed: OK
I20240503 07:59:15.385699 9030 rebalancer_tool.cc:1074] tablet 09866b93f75545a19c9451556254da48: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:15.399915 9030 rebalancer_tool.cc:1074] tablet c597bc1a19274f55b9d60558db35fcae: 'a97073f89fd44702bccb74a9b76f858d' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:15.486433 9030 rebalancer_tool.cc:1136] tablet 60f643dce3eb4c669b36b0714a31651b: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:15.530176 9030 rebalancer_tool.cc:1074] tablet 8861b52c306e43d8b0615202cf7d8e35: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:15.634739 9030 rebalancer_tool.cc:1136] tablet 39b01318dc1543cba2cc991e9db91c28: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.678766 9030 rebalancer_tool.cc:1074] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:15.778195 9030 rebalancer_tool.cc:1136] tablet bfd4259777e5492c904e6d65a36d92ed: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:15.948845 9030 rebalancer_tool.cc:1136] tablet f635e30fb6974fa1a79d8a46eecc3d5e: '3d08a06d53b14381bee6223c4e1a6027' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:15.964808 9030 rebalancer_tool.cc:1136] tablet 2d3f03610866454595f4b6b87b7cd9cb: 'c167a0a433d34b74affed13cbf96e302' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:15.993288 9030 rebalancer_tool.cc:1074] tablet d4308bc87be24f28b6566ce76cae6e50: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move scheduled
I20240503 07:59:16.083006 9030 rebalancer_tool.cc:1136] tablet f4e4ca6638764fb49f3afcab7688e2b3: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.089006 9030 rebalancer_tool.cc:1136] tablet 5060df68b95e48a8a849e77ba926e608: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:16.137351 9030 rebalancer_tool.cc:1074] tablet afeff5ce33bc4205a8599bbb66b24672: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:16.163012 9030 rebalancer_tool.cc:1074] tablet a833fb4980c04673a335d020a3ad3dd3: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:16.567724 9030 rebalancer_tool.cc:1136] tablet 558c0eb4a4334dbe91ac60fad4e820dc: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '3d08a06d53b14381bee6223c4e1a6027' move completed: OK
I20240503 07:59:16.583222 9030 rebalancer_tool.cc:1136] tablet 5631f3e97fe04013a2a7140f44260003: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:16.673249 9030 rebalancer_tool.cc:1074] tablet 435f73fb9f384b10b353df19037bcac3: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:16.699066 9030 rebalancer_tool.cc:1074] tablet b61209b73f1f4872b5eccc823b1fa3df: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:16.781275 9030 rebalancer_tool.cc:1136] tablet c597bc1a19274f55b9d60558db35fcae: 'a97073f89fd44702bccb74a9b76f858d' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:16.804905 9030 rebalancer_tool.cc:1136] tablet 551f25dcee034b7894498e3c748b0f0e: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.811823 9030 rebalancer_tool.cc:1136] tablet c8cf3d6a3ace46769854ed61f340fb25: '616fdda6ba714cb7be01a564de2b9db2' -> 'c167a0a433d34b74affed13cbf96e302' move completed: OK
I20240503 07:59:16.820207 9030 rebalancer_tool.cc:1136] tablet c5b4c776eac24f739d85e09707cd751b: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:16.832350 9030 rebalancer_tool.cc:1136] tablet e103e0e9cb16416dba7f8bbfc08ceac8: '3d08a06d53b14381bee6223c4e1a6027' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:16.862488 9030 rebalancer_tool.cc:1074] tablet edf2c780d0d64db79dca88eac77d4478: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move scheduled
I20240503 07:59:16.882623 9030 rebalancer_tool.cc:1074] tablet afcd161598db41d7962caca61f63ef77: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:16.926805 9030 rebalancer_tool.cc:1136] tablet 974c815ff8e94190a7bf28444cb3cd9f: 'a97073f89fd44702bccb74a9b76f858d' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:16.964197 9030 rebalancer_tool.cc:1136] tablet 82de209205b342cfaac10506fdac9303: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:16.993131 9030 rebalancer_tool.cc:1074] tablet 9772b487ca864ad98e9d79d8435c34c6: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:17.077139 9030 rebalancer_tool.cc:1136] tablet 2a10085c34be4d31aae9f56a2ec1ba23: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:17.676190 9030 rebalancer_tool.cc:1136] tablet afeff5ce33bc4205a8599bbb66b24672: '616fdda6ba714cb7be01a564de2b9db2' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:17.704397 9030 rebalancer_tool.cc:1136] tablet d4308bc87be24f28b6566ce76cae6e50: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
I20240503 07:59:17.742390 9030 rebalancer_tool.cc:1136] tablet a833fb4980c04673a335d020a3ad3dd3: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:18.305913 9030 rebalancer_tool.cc:1136] tablet b61209b73f1f4872b5eccc823b1fa3df: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move completed: OK
I20240503 07:59:18.321419 9030 rebalancer_tool.cc:1136] tablet 435f73fb9f384b10b353df19037bcac3: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.357174 9030 rebalancer_tool.cc:1136] tablet edf2c780d0d64db79dca88eac77d4478: 'ad910f92ef9d4db8a8962cecba289f8a' -> '616fdda6ba714cb7be01a564de2b9db2' move completed: OK
I20240503 07:59:18.371577 9030 rebalancer_tool.cc:1136] tablet 9772b487ca864ad98e9d79d8435c34c6: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.623713 9030 rebalancer_tool.cc:1136] tablet afcd161598db41d7962caca61f63ef77: 'd1f36d57661b4429a4be8d10b58d8ff7' -> '9553fe3c4ccd4e91b7c176aad82fec09' move completed: OK
I20240503 07:59:18.636446 9030 rebalancer_tool.cc:1136] tablet 09866b93f75545a19c9451556254da48: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'ad910f92ef9d4db8a8962cecba289f8a' move completed: OK
I20240503 07:59:18.849134 9030 rebalancer_tool.cc:1136] tablet 8861b52c306e43d8b0615202cf7d8e35: 'a97073f89fd44702bccb74a9b76f858d' -> 'c8d71787690245e681c85a087086e7ed' move completed: OK
W20240503 07:59:19.076227 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8f467f94f8014d3e979398f9ea0857e3: could not find any suitable replica to move from server c8d71787690245e681c85a087086e7ed to server ad910f92ef9d4db8a8962cecba289f8a
W20240503 07:59:19.076323 9030 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 68cd74d1f907476d8cccfd32a1339623: could not find any suitable replica to move from server 9553fe3c4ccd4e91b7c176aad82fec09 to server c8d71787690245e681c85a087086e7ed
I20240503 07:59:19.096592 9030 rebalancer_tool.cc:1074] tablet 5b745ba777b2476fbae78706b35125f6: 'c8d71787690245e681c85a087086e7ed' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.112599 9030 rebalancer_tool.cc:1074] tablet edaf4aa5285f49c297ab412ceb179be9: '3d08a06d53b14381bee6223c4e1a6027' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.126675 9030 rebalancer_tool.cc:1074] tablet 60f643dce3eb4c669b36b0714a31651b: 'ad910f92ef9d4db8a8962cecba289f8a' -> '9553fe3c4ccd4e91b7c176aad82fec09' move scheduled
I20240503 07:59:19.139557 9030 rebalancer_tool.cc:1074] tablet e1b2ea716ca64e309efccd291db503d6: 'c8d71787690245e681c85a087086e7ed' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.154740 9030 rebalancer_tool.cc:1074] tablet b61209b73f1f4872b5eccc823b1fa3df: '616fdda6ba714cb7be01a564de2b9db2' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.167088 9030 rebalancer_tool.cc:1074] tablet f7442d12e0234db38d896c09a9e73162: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.192380 9030 rebalancer_tool.cc:1074] tablet cfd4deb6a5104141bfa51fe4114bb708: 'ad910f92ef9d4db8a8962cecba289f8a' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.212666 9030 rebalancer_tool.cc:1074] tablet 551f25dcee034b7894498e3c748b0f0e: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.228022 9030 rebalancer_tool.cc:1074] tablet 3e889be4506e4f01a978f9e94f6b18d2: '616fdda6ba714cb7be01a564de2b9db2' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:19.245142 9030 rebalancer_tool.cc:1074] tablet 00bcbd5440ff4d56b1975edc4041b428: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
I20240503 07:59:19.258183 9030 rebalancer_tool.cc:1074] tablet fc7d26466129405bbe6db2721823bae1: '9553fe3c4ccd4e91b7c176aad82fec09' -> 'd1f36d57661b4429a4be8d10b58d8ff7' move scheduled
I20240503 07:59:19.274138 9030 rebalancer_tool.cc:1074] tablet fa8b855a6985404da6bd34daa0eb9ae7: '616fdda6ba714cb7be01a564de2b9db2' -> 'a97073f89fd44702bccb74a9b76f858d' move scheduled
I20240503 07:59:19.287663 9030 rebalancer_tool.cc:1074] tablet 2054e7790d7745c2a1b08e903c64d1b7: '3d08a06d53b14381bee6223c4e1a6027' -> 'ad910f92ef9d4db8a8962cecba289f8a' move scheduled
I20240503 07:59:19.300256 9030 rebalancer_tool.cc:1074] tablet 8c01e3ce57874e24ae75a41243fdeaaf: 'd1f36d57661b4429a4be8d10b58d8ff7' -> 'c167a0a433d34b74affed13cbf96e302' move scheduled
W20240503 07:59:19.934450 13800 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:19.739648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:40925 (local address 127.0.0.1:34174)
0503 07:59:19.739916 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:19.739930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0503 07:59:19.740007 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 07:59:19.933892 (+193885us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20240503 07:59:19.934860 9030 rebalancer_tool.cc:1123] tablet 00bcbd5440ff4d56b1975edc4041b428: '3d08a06d53b14381bee6223c4e1a6027' -> 'c167a0a433d34b74affed13cbf96e302' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 07:59:19.955201 9030 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): unacceptable health status UNAVAILABLE
I20240503 07:59:20.194370 6298 raft_consensus.cc:1203] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 07:59:20.195957 12930 consensus_queue.cc:1035] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.199775 13924 consensus_queue.cc:1035] T 30616a5a684b4cfda26ad3543f11b07c P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.211159 6690 raft_consensus.cc:1203] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 07:59:20.217242 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:20.221431 13872 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.221855 13381 tablet.cc:1722] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:20.225920 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47] alter complete (version 3)
I20240503 07:59:20.286144 6430 raft_consensus.cc:1203] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.293671 13175 consensus_queue.cc:1035] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.294214 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.296891 9347 consensus_queue.cc:1035] T acdb48f75a014b8d9e82f045950f4c98 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.305990 6235 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.324342 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:20.329777 13018 raft_consensus.cc:1050] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.329859 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.330111 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.330183 13018 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.330453 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.330687 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.331580 8842 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.332046 6690 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.332415 12608 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.332545 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 7af0459cb687453eb98aa1db18f39828 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.332702 12608 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.332820 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7af0459cb687453eb98aa1db18f39828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.333070 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6db695ea2a48c49e687d95df2af58d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.333298 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6db695ea2a48c49e687d95df2af58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335070 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.335263 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335286 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.335476 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.335592 13933 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:20.336270 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6db695ea2a48c49e687d95df2af58d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.336483 13933 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.336489 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c6db695ea2a48c49e687d95df2af58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.337293 8976 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.337785 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 323603da9f964bee9c8b565a20fa51a5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=b247ad7d2d5a4d4f819253f7b501c0dc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:20.338042 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323603da9f964bee9c8b565a20fa51a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.338182 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.338280 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:20.339882 13938 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:20.340415 13934 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.341400 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet dbd4400894d14f839670d209468f1929
I20240503 07:59:20.341555 13940 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:20.342224 6690 raft_consensus.cc:1203] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.338757 13932 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.342401 13934 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.342414 13940 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.343366 13938 tablet_bootstrap.cc:654] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.344312 13934 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.344457 13934 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:20.344707 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd4400894d14f839670d209468f1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.345022 13934 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.345141 13934 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.345172 13934 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.345273 13934 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.345364 6151 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.345580 13934 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.345688 13938 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:20.345763 13938 ts_tablet_manager.cc:1367] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:20.345793 13934 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.346940 13934 tablet_bootstrap.cc:654] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.347182 13938 raft_consensus.cc:348] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.347308 13938 raft_consensus.cc:374] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.347371 13938 raft_consensus.cc:729] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.347497 13938 consensus_queue.cc:260] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.347965 13933 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:20.348047 13933 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 07:59:20.348310 13938 ts_tablet_manager.cc:1398] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.348453 13933 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.348531 13933 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.348558 13933 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.348587 13931 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.348660 13933 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.348809 8842 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.348946 13933 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.349093 13933 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:20.350208 13933 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.350293 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd4400894d14f839670d209468f1929 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:20 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.350524 13946 tablet_replica.cc:330] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.350557 6560 consensus_queue.cc:237] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.350630 13946 raft_consensus.cc:2229] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:20.350809 13946 raft_consensus.cc:2258] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.352063 8842 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.352074 6691 raft_consensus.cc:1261] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.352434 13018 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.352579 13931 consensus_queue.cc:1035] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.353041 13946 ts_tablet_manager.cc:1875] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.353053 8976 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.353289 6691 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.354890 13946 ts_tablet_manager.cc:1888] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.354961 13946 log.cc:1196] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/dbd4400894d14f839670d209468f1929
I20240503 07:59:20.355191 13946 ts_tablet_manager.cc:1909] T dbd4400894d14f839670d209468f1929 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:20.355448 13940 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:20.355525 13940 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 07:59:20.355852 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47]) successfully deleted
I20240503 07:59:20.356029 13940 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.356185 13940 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.356254 13940 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.356400 13940 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.356945 13018 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.357333 13940 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.357839 13933 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:20.357892 13933 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:20.358237 13933 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.358320 13933 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.358366 13933 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.358458 13933 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.358686 13933 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.358626 8842 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.360541 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet edaf4aa5285f49c297ab412ceb179be9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.360965 6155 catalog_manager.cc:5484] T edaf4aa5285f49c297ab412ceb179be9 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.363124 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet edaf4aa5285f49c297ab412ceb179be9 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:20.363369 6690 raft_consensus.cc:2932] T edaf4aa5285f49c297ab412ceb179be9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.365797 13944 tablet_replica.cc:330] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.366036 13944 raft_consensus.cc:2229] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.366226 13944 raft_consensus.cc:2258] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.366912 13944 ts_tablet_manager.cc:1875] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:20.367380 13937 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:20.367580 13934 tablet_bootstrap.cc:492] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.367650 13934 ts_tablet_manager.cc:1367] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240503 07:59:20.368002 13934 raft_consensus.cc:348] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.368075 13934 raft_consensus.cc:374] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.368095 13934 raft_consensus.cc:729] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.368146 13937 tablet_bootstrap.cc:654] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.368191 13934 consensus_queue.cc:260] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.368471 13934 ts_tablet_manager.cc:1398] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.368975 13949 tablet_replica.cc:330] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:20.369252 13937 tablet_bootstrap.cc:492] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:20.369338 13937 ts_tablet_manager.cc:1367] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:20.369750 13937 raft_consensus.cc:348] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.369877 13937 raft_consensus.cc:374] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.369932 13937 raft_consensus.cc:729] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.370080 13937 consensus_queue.cc:260] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.370385 13937 ts_tablet_manager.cc:1398] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.370559 13937 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:20.370682 13944 ts_tablet_manager.cc:1888] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:20.370772 13944 log.cc:1196] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/dbd4400894d14f839670d209468f1929
I20240503 07:59:20.369123 13949 raft_consensus.cc:2229] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.371003 13949 raft_consensus.cc:2258] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.371038 13944 ts_tablet_manager.cc:1909] T dbd4400894d14f839670d209468f1929 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:20.371366 13937 tablet_bootstrap.cc:654] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.371481 13949 ts_tablet_manager.cc:1875] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.371806 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet dbd4400894d14f839670d209468f1929 (table rebalancer_dynamic_table_new_name [id=a3f9b29ab4d44204a1b074d6db58cc47]) successfully deleted
I20240503 07:59:20.373229 13937 tablet_bootstrap.cc:492] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:20.373301 13937 ts_tablet_manager.cc:1367] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.373664 13937 raft_consensus.cc:348] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.373791 13937 raft_consensus.cc:374] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.373833 13937 raft_consensus.cc:729] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.373945 13937 consensus_queue.cc:260] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.374164 13937 ts_tablet_manager.cc:1398] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:20.374256 13949 ts_tablet_manager.cc:1888] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.374327 13949 log.cc:1196] T edaf4aa5285f49c297ab412ceb179be9 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/edaf4aa5285f49c297ab412ceb179be9
I20240503 07:59:20.374900 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet edaf4aa5285f49c297ab412ceb179be9 (table rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]) successfully deleted
I20240503 07:59:20.379088 6560 raft_consensus.cc:1203] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 07:59:20.434762 13478 raft_consensus.cc:1050] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.435117 13478 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.436311 6296 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.436849 6430 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.436895 8575 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.437242 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.437463 13478 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.437633 12241 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.440061 13478 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.440352 6296 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.441825 6430 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.442483 8575 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.442780 6150 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.445134 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:20.446187 8842 consensus_queue.cc:237] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.447207 6430 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.447564 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.447764 6296 raft_consensus.cc:1261] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.448927 13953 consensus_queue.cc:1035] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.450551 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.450567 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.450767 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.450789 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.451437 11508 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.452710 13940 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:20.453051 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca739ac52346ae81c06834fbb781ee (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]), partition=
I20240503 07:59:20.453086 6430 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.453255 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca739ac52346ae81c06834fbb781ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:20.453716 13940 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.454921 13940 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:20.455000 13940 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:20.454968 6156 catalog_manager.cc:5484] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.455319 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b745ba777b2476fbae78706b35125f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.455411 13940 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.455504 13940 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.455528 13940 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.455619 13940 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.455848 13940 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.456022 13934 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:20.456861 13934 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.459112 6296 raft_consensus.cc:2932] T 5b745ba777b2476fbae78706b35125f6 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.459249 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b745ba777b2476fbae78706b35125f6 with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.459962 13944 tablet_replica.cc:330] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.460081 13944 raft_consensus.cc:2229] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.460191 13944 raft_consensus.cc:2258] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.460227 13957 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:20.461330 13934 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:20.461408 13934 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:20.461819 13934 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.461923 13934 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.461962 13934 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.462072 13934 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.462162 13957 tablet_bootstrap.cc:654] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:20.462365 13934 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.463126 13957 tablet_bootstrap.cc:492] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:20.463204 13957 ts_tablet_manager.cc:1367] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:20.463586 13957 raft_consensus.cc:348] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.463682 13957 raft_consensus.cc:374] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:20.463711 13957 raft_consensus.cc:729] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:20.463827 13957 consensus_queue.cc:260] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.464115 13957 ts_tablet_manager.cc:1398] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:20.465276 13944 ts_tablet_manager.cc:1875] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.466995 13944 ts_tablet_manager.cc:1888] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.467056 13944 log.cc:1196] T 5b745ba777b2476fbae78706b35125f6 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5b745ba777b2476fbae78706b35125f6
I20240503 07:59:20.467489 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 5b745ba777b2476fbae78706b35125f6 (table rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]) successfully deleted
I20240503 07:59:20.502321 13180 raft_consensus.cc:1050] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.502652 13180 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.503777 8842 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.503865 8575 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.504230 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.504706 11524 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.505007 6430 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.505391 13924 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.506706 13180 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.507134 8575 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.508638 8842 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.509641 6430 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.511090 6151 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.515919 8976 consensus_queue.cc:237] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.516875 8842 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.516907 6430 raft_consensus.cc:1261] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.517211 12930 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.517400 13924 consensus_queue.cc:1035] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.518800 13180 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.519253 6430 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.519253 8842 raft_consensus.cc:2932] T e1b2ea716ca64e309efccd291db503d6 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.520179 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e1b2ea716ca64e309efccd291db503d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.521622 6156 catalog_manager.cc:5484] T e1b2ea716ca64e309efccd291db503d6 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: config changed from index 3 to 4, VOTER c8d71787690245e681c85a087086e7ed (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.523594 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b2ea716ca64e309efccd291db503d6 with delete_type TABLET_DATA_TOMBSTONED (TS c8d71787690245e681c85a087086e7ed not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:20.523761 13944 tablet_replica.cc:330] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:20.523897 13944 raft_consensus.cc:2229] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.524039 13944 raft_consensus.cc:2258] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.524587 13944 ts_tablet_manager.cc:1875] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.526121 13944 ts_tablet_manager.cc:1888] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.526182 13944 log.cc:1196] T e1b2ea716ca64e309efccd291db503d6 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/e1b2ea716ca64e309efccd291db503d6
I20240503 07:59:20.526561 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet e1b2ea716ca64e309efccd291db503d6 (table rebalancer_extra_table_12 [id=8de878c3d0964066a86c2aacee0a7bc0]) successfully deleted
I20240503 07:59:20.557751 9347 raft_consensus.cc:1050] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:20.558122 9347 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.559278 6560 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559363 6430 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559533 8842 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.559859 13969 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.560066 13175 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.560237 9347 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.562860 13175 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.563524 13953 raft_consensus.cc:1050] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.563387 6560 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.563907 13953 consensus_queue.cc:237] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.564797 6430 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.566638 8842 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.566803 6155 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:59:20.568413 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.569042 6430 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.569497 12241 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.570585 6690 consensus_queue.cc:237] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.571182 6560 raft_consensus.cc:1261] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.571561 12241 consensus_queue.cc:1035] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.571753 6428 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.572259 6559 raft_consensus.cc:1261] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.572535 13278 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.572952 13278 consensus_queue.cc:1035] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.573176 11508 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.573779 13924 raft_consensus.cc:482] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.573881 13924 raft_consensus.cc:504] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.574262 13924 leader_election.cc:288] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.574182 6559 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.575327 6428 raft_consensus.cc:2932] T cfd4deb6a5104141bfa51fe4114bb708 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
W20240503 07:59:20.575553 8913 leader_election.cc:334] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.575577 13969 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.577349 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6db695ea2a48c49e687d95df2af58d" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:20.577457 6690 raft_consensus.cc:2454] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.577519 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e889be4506e4f01a978f9e94f6b18d2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.577940 8912 leader_election.cc:302] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.578177 13924 raft_consensus.cc:2781] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.578231 13924 raft_consensus.cc:482] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.578266 13924 raft_consensus.cc:3037] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.578924 6151 catalog_manager.cc:5484] T 3e889be4506e4f01a978f9e94f6b18d2 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.579200 13924 raft_consensus.cc:504] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.579521 13924 leader_election.cc:288] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.579802 6156 catalog_manager.cc:5484] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.580447 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c6db695ea2a48c49e687d95df2af58d" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:20.580543 6690 raft_consensus.cc:3037] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.581406 6690 raft_consensus.cc:2454] T 2c6db695ea2a48c49e687d95df2af58d P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
W20240503 07:59:20.581733 8913 leader_election.cc:334] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.582398 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e889be4506e4f01a978f9e94f6b18d2 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.582432 8912 leader_election.cc:302] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.582573 13946 tablet_replica.cc:330] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.582473 6430 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.582691 13946 raft_consensus.cc:2229] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.582806 13946 raft_consensus.cc:2258] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.582782 6559 raft_consensus.cc:2932] T 3e889be4506e4f01a978f9e94f6b18d2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.583699 13946 ts_tablet_manager.cc:1875] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.584306 12434 raft_consensus.cc:482] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.584393 12434 raft_consensus.cc:504] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.584636 13924 raft_consensus.cc:2781] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.584707 13924 raft_consensus.cc:686] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.584793 12434 leader_election.cc:288] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.584841 13924 consensus_queue.cc:237] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.586316 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:20.586436 8575 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:20.586658 6363 leader_election.cc:302] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.586779 13946 ts_tablet_manager.cc:1888] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.586823 12434 raft_consensus.cc:2781] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.586864 12434 raft_consensus.cc:482] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.586856 13946 log.cc:1196] T 3e889be4506e4f01a978f9e94f6b18d2 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3e889be4506e4f01a978f9e94f6b18d2
I20240503 07:59:20.586897 12434 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.587584 6151 catalog_manager.cc:5484] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.587814 12434 raft_consensus.cc:504] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.589607 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 3e889be4506e4f01a978f9e94f6b18d2 (table rebalancer_extra_table_20 [id=7d2ea3796b2745cc82fc27822f80cda4]) successfully deleted
I20240503 07:59:20.589951 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:20.590058 8842 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:20.590971 12434 leader_election.cc:288] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.591313 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:20.591410 8842 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.592053 8842 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:20.592119 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323603da9f964bee9c8b565a20fa51a5" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:20.592221 8575 raft_consensus.cc:3037] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.592933 6365 leader_election.cc:302] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.593142 8575 raft_consensus.cc:2454] T 323603da9f964bee9c8b565a20fa51a5 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:20.593152 12434 raft_consensus.cc:2781] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.593230 12434 raft_consensus.cc:686] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:20.593353 12434 consensus_queue.cc:237] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:20.596401 6155 catalog_manager.cc:5484] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.688905 12344 raft_consensus.cc:1050] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.689313 12344 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.690642 6690 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.690685 6296 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.691016 8976 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.691051 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.691236 13931 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.691416 12344 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.693271 12344 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.694906 6296 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.695106 6690 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.695760 8976 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.696202 6154 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: config changed from index 2 to 3, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.697816 13953 raft_consensus.cc:1050] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2: attempting to promote NON_VOTER d1f36d57661b4429a4be8d10b58d8ff7 to VOTER
I20240503 07:59:20.698227 13953 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.699641 6430 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEARNER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.699788 8975 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.700366 11508 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.700573 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.700747 6559 consensus_queue.cc:237] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.701951 6298 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 07:59:20.702157 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.702311 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.702453 6690 raft_consensus.cc:1261] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.703161 13018 consensus_queue.cc:1035] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.704679 12608 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.705197 6690 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.706224 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f7442d12e0234db38d896c09a9e73162 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.707425 6298 raft_consensus.cc:2932] T f7442d12e0234db38d896c09a9e73162 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.707638 6151 catalog_manager.cc:5484] T f7442d12e0234db38d896c09a9e73162 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.710460 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet f7442d12e0234db38d896c09a9e73162 with delete_type TABLET_DATA_TOMBSTONED (TS 3d08a06d53b14381bee6223c4e1a6027 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:20.710639 13949 tablet_replica.cc:330] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:20.710759 13949 raft_consensus.cc:2229] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.710891 13949 raft_consensus.cc:2258] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.711412 13949 ts_tablet_manager.cc:1875] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.711786 11508 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.712185 6430 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.713438 13949 ts_tablet_manager.cc:1888] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.713511 13949 log.cc:1196] T f7442d12e0234db38d896c09a9e73162 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f7442d12e0234db38d896c09a9e73162
I20240503 07:59:20.713527 8975 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.714766 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet f7442d12e0234db38d896c09a9e73162 (table rebalancer_extra_table_18 [id=34387136ecf7491f8bb4e20130cc6840]) successfully deleted
I20240503 07:59:20.715384 6154 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 2 to 3, d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.717758 8842 consensus_queue.cc:237] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } }
I20240503 07:59:20.718735 6430 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.718845 8975 raft_consensus.cc:1261] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.719154 12241 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.719332 11508 consensus_queue.cc:1035] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.720535 13478 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.721140 6430 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
I20240503 07:59:20.722402 6143 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc7d26466129405bbe6db2721823bae1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.722788 8975 raft_consensus.cc:2932] T fc7d26466129405bbe6db2721823bae1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } }
W20240503 07:59:20.724363 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.724944 6155 catalog_manager.cc:5484] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: config changed from index 3 to 4, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.731709 11508 raft_consensus.cc:482] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.731782 11508 raft_consensus.cc:504] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.732064 11508 leader_election.cc:288] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.732290 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:20.732384 6430 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:20.732378 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:20.732482 8975 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:20.732666 8779 leader_election.cc:302] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.732861 11508 raft_consensus.cc:2781] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.732918 11508 raft_consensus.cc:482] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.732944 11508 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.733557 11508 raft_consensus.cc:504] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.733826 11508 leader_election.cc:288] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.734059 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:20.734153 6430 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.734298 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af0459cb687453eb98aa1db18f39828" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:20.734400 8975 raft_consensus.cc:3037] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.734916 6430 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:20.735265 8975 raft_consensus.cc:2454] T 7af0459cb687453eb98aa1db18f39828 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:20.735242 8779 leader_election.cc:302] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:20.735502 11508 raft_consensus.cc:2781] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.735589 11508 raft_consensus.cc:686] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:20.735693 11508 consensus_queue.cc:237] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.737213 6155 catalog_manager.cc:5484] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:20.741977 11524 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.742059 11524 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:20.742349 11524 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:20.742694 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:20.743294 8913 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.743371 8913 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:20.743486 11524 raft_consensus.cc:2726] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.771991 13248 raft_consensus.cc:1050] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:20.772346 13248 consensus_queue.cc:237] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.773669 6298 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.773728 8976 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.774058 13114 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.774154 6559 raft_consensus.cc:1261] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 07:59:20.774284 13989 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.774533 13248 consensus_queue.cc:1035] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:20.776130 13114 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.776551 8976 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.776528 6559 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.779863 6155 catalog_manager.cc:5484] T 8c01e3ce57874e24ae75a41243fdeaaf P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.780011 6298 raft_consensus.cc:2932] T 8c01e3ce57874e24ae75a41243fdeaaf P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:20.785701 11524 raft_consensus.cc:482] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:20.785775 11524 raft_consensus.cc:504] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.786214 11524 leader_election.cc:288] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:20.786530 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:20.786571 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:20.786657 8575 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.786659 6559 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:20.786952 8910 leader_election.cc:302] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:20.787107 11524 raft_consensus.cc:2781] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:20.787160 11524 raft_consensus.cc:482] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:20.787185 11524 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.787981 13927 raft_consensus.cc:1050] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.787978 11524 raft_consensus.cc:504] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.788236 11524 leader_election.cc:288] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:20.788368 13927 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.788605 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:20.788695 6559 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.789575 6559 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.789598 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca739ac52346ae81c06834fbb781ee" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:20.789690 8575 raft_consensus.cc:3037] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:20.789834 6690 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:20.790241 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 07:59:20.790225 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:20.790460 8575 raft_consensus.cc:2454] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.790611 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:20.791088 8910 leader_election.cc:302] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:20.791489 11524 raft_consensus.cc:2781] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:20.791563 11524 raft_consensus.cc:686] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.791688 11524 consensus_queue.cc:237] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
W20240503 07:59:20.793618 6235 consensus_peers.cc:485] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.793823 6155 catalog_manager.cc:5484] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.794114 13927 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.795707 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.796370 6690 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.797303 6155 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 5 to 6, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.800063 6298 consensus_queue.cc:237] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.801074 6430 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:20.801110 6690 raft_consensus.cc:1261] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 07:59:20.801430 13537 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:20.801560 13979 consensus_queue.cc:1035] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:20.803009 13927 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.803488 6690 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.803580 6430 raft_consensus.cc:2932] T 551f25dcee034b7894498e3c748b0f0e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.804426 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551f25dcee034b7894498e3c748b0f0e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.805763 6154 catalog_manager.cc:5484] T 551f25dcee034b7894498e3c748b0f0e P c167a0a433d34b74affed13cbf96e302 reported cstate change: config changed from index 6 to 7, VOTER 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.829605 13924 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.829686 13924 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:20.830013 13924 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.830281 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
W20240503 07:59:20.831126 8913 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.831203 8913 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302
I20240503 07:59:20.831327 13924 raft_consensus.cc:2726] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.869983 13927 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.870079 13927 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:20.870412 13927 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:20.870725 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:20.871343 6235 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.871412 6235 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:20.871526 13927 raft_consensus.cc:2726] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.898016 12241 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.898119 12241 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.898483 12241 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:20.898789 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
W20240503 07:59:20.899399 8779 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.899472 8779 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2; no voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.899595 12241 raft_consensus.cc:2726] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.901401 13924 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.901476 13924 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.901738 13924 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.902022 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:20.902135 8842 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:20.902398 8912 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:20.902539 13924 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:20.902577 13924 raft_consensus.cc:482] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
W20240503 07:59:20.902583 8913 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.902630 13924 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:20.903455 13924 raft_consensus.cc:504] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.903729 13924 leader_election.cc:288] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:20.903985 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873c7786052b4b9c97cc0c1bcb6e7a03" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:20.904080 8842 raft_consensus.cc:3037] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:20.904909 8913 leader_election.cc:334] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.905005 8842 raft_consensus.cc:2454] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 2.
I20240503 07:59:20.905283 8912 leader_election.cc:302] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.905438 13924 raft_consensus.cc:2781] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:20.905537 13924 raft_consensus.cc:686] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:20.905661 13924 consensus_queue.cc:237] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:20.907416 13961 raft_consensus.cc:1050] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:20.907418 13931 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.907503 13931 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.907785 13961 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.907841 13931 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:20.908463 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
W20240503 07:59:20.908823 6497 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.908799 6151 catalog_manager.cc:5484] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.909051 6496 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a; no voters: 3d08a06d53b14381bee6223c4e1a6027, 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:20.909209 6430 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.909253 13931 raft_consensus.cc:2726] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:20.909530 6690 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.909891 13922 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.910182 13922 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.910250 8842 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.910559 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.912223 13922 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.912853 8842 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.913064 13278 raft_consensus.cc:1050] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d: attempting to promote NON_VOTER ad910f92ef9d4db8a8962cecba289f8a to VOTER
I20240503 07:59:20.913400 13278 consensus_queue.cc:237] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:20.914009 6430 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.914420 6690 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.914544 6559 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEARNER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.915017 13969 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 07:59:20.915385 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.916620 8976 raft_consensus.cc:1261] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 07:59:20.917002 13175 consensus_queue.cc:1035] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:20.917230 6151 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 3, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.918489 13969 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.920024 8575 consensus_queue.cc:237] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:20.920230 6559 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.920262 8976 raft_consensus.cc:2932] T 2054e7790d7745c2a1b08e903c64d1b7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } } }
I20240503 07:59:20.921972 6430 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.922493 6690 raft_consensus.cc:1261] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 07:59:20.922859 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.923035 6154 catalog_manager.cc:5484] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 2 to 3, ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:20.924599 13999 consensus_queue.cc:1035] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:20.924958 12035 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.926461 6141 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa8b855a6985404da6bd34daa0eb9ae7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 07:59:20.926913 6690 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.927008 6430 raft_consensus.cc:2932] T fa8b855a6985404da6bd34daa0eb9ae7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.927420 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:20.929700 6156 catalog_manager.cc:5484] T fa8b855a6985404da6bd34daa0eb9ae7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: config changed from index 3 to 4, VOTER 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:20.930646 6559 raft_consensus.cc:1261] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.930752 8575 raft_consensus.cc:1261] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:20.931006 13924 consensus_queue.cc:1035] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.931244 13924 consensus_queue.cc:1035] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:20.932912 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8b855a6985404da6bd34daa0eb9ae7 with delete_type TABLET_DATA_TOMBSTONED (TS 616fdda6ba714cb7be01a564de2b9db2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:20.933099 13946 tablet_replica.cc:330] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:20.933223 13946 raft_consensus.cc:2229] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:20.933337 13946 raft_consensus.cc:2258] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:20.933940 13946 ts_tablet_manager.cc:1875] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 07:59:20.935243 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.935534 14012 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:20.936138 13946 ts_tablet_manager.cc:1888] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 07:59:20.936213 13946 log.cc:1196] T fa8b855a6985404da6bd34daa0eb9ae7 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/fa8b855a6985404da6bd34daa0eb9ae7
I20240503 07:59:20.936728 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet fa8b855a6985404da6bd34daa0eb9ae7 (table rebalancer_extra_table_14 [id=8f467f94f8014d3e979398f9ea0857e3]) successfully deleted
I20240503 07:59:20.937454 6151 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.938338 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.939792 14013 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:20.939714 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:20.940440 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.940490 14014 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:20.941284 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63]). Expected version 1 got 0
I20240503 07:59:20.941386 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 1)
W20240503 07:59:20.945673 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fc7d26466129405bbe6db2721823bae1: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.978050 12930 consensus_queue.cc:1035] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:20.981268 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:20.994494 13999 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:20.994580 13999 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:20.994890 13999 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:20.995206 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
W20240503 07:59:20.995909 8517 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:20.995981 8517 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c167a0a433d34b74affed13cbf96e302
I20240503 07:59:20.996090 13999 raft_consensus.cc:2726] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.037695 12241 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.037802 12241 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.038144 12241 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:21.038590 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
W20240503 07:59:21.039026 8779 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.039093 8779 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:21.039198 12241 raft_consensus.cc:2726] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.045714 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:21.050981 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.051134 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.051265 14014 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:21.051818 14012 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.051966 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.052238 14013 tablet.cc:1620] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.057188 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 2)
I20240503 07:59:21.058388 13989 consensus_queue.cc:1035] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.061854 12434 consensus_queue.cc:1035] T 323603da9f964bee9c8b565a20fa51a5 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.070039 13999 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.070134 13999 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.070643 13999 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:21.070969 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:21.071100 6298 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
W20240503 07:59:21.071451 8517 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.071538 8514 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.071683 13999 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.071743 13999 raft_consensus.cc:482] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.071794 13999 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.072598 13999 raft_consensus.cc:504] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.072947 13999 leader_election.cc:288] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:21.073240 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f635e30fb6974fa1a79d8a46eecc3d5e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:21.073343 6298 raft_consensus.cc:3037] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.073772 8517 leader_election.cc:334] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.074239 6298 raft_consensus.cc:2454] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.074530 8514 leader_election.cc:302] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.074687 13999 raft_consensus.cc:2781] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.074787 13999 raft_consensus.cc:686] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.074936 13999 consensus_queue.cc:237] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } }
I20240503 07:59:21.076658 6156 catalog_manager.cc:5484] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:21.081722 13018 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.081808 13018 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.082123 13018 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.082468 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
W20240503 07:59:21.083017 6497 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.083072 6497 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a; no voters: 9553fe3c4ccd4e91b7c176aad82fec09, c8d71787690245e681c85a087086e7ed
I20240503 07:59:21.083176 13018 raft_consensus.cc:2726] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 07:59:21.107744 13999 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.107829 13999 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.108155 13999 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.108433 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:21.108556 6559 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.108816 8514 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.108989 13999 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.109041 13999 raft_consensus.cc:482] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.109077 13999 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.109153 8517 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.109843 13999 raft_consensus.cc:504] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.110270 13999 leader_election.cc:288] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.110412 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f534eefb7c334889a9c5d02175ae2d94" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:21.110517 6559 raft_consensus.cc:3037] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.110992 8517 leader_election.cc:334] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.111217 6559 raft_consensus.cc:2454] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.111431 8514 leader_election.cc:302] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.111565 13999 raft_consensus.cc:2781] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.111644 13999 raft_consensus.cc:686] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.111766 13999 consensus_queue.cc:237] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.113495 6156 catalog_manager.cc:5484] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.150990 12434 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.151103 12434 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.151455 12434 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.151723 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:21.151861 8842 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:21.152105 6365 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:21.152350 12434 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.152427 12434 raft_consensus.cc:482] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.152470 12434 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.152621 6366 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.153277 12434 raft_consensus.cc:504] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.153576 12434 leader_election.cc:288] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.153829 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b61209b73f1f4872b5eccc823b1fa3df" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:21.153975 8843 raft_consensus.cc:3037] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.154484 6366 leader_election.cc:334] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.154873 8843 raft_consensus.cc:2454] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 2.
I20240503 07:59:21.155133 6365 leader_election.cc:302] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.155299 12434 raft_consensus.cc:2781] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.155402 12434 raft_consensus.cc:686] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:21.155529 12434 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:21.157403 6156 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.162391 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:21.164655 13927 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.164752 13927 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.165184 13927 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.165781 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.165892 8975 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:21.166375 6234 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:21.166649 13927 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.166699 13927 raft_consensus.cc:482] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.166733 13927 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.166934 14012 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.167603 13927 raft_consensus.cc:504] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.167901 13927 leader_election.cc:288] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.168177 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f260c4c4e7e4fcc8fbce11064bdc338" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.168277 8975 raft_consensus.cc:3037] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.168841 14014 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.169121 8975 raft_consensus.cc:2454] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 2.
W20240503 07:59:21.169867 6235 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.170441 6235 leader_election.cc:334] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.171401 6234 leader_election.cc:302] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.171775 14013 tablet.cc:1722] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.172427 13927 raft_consensus.cc:2781] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.172518 13927 raft_consensus.cc:686] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [term 2 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:21.172649 13927 consensus_queue.cc:237] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.172966 13979 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.173041 13979 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.173383 13979 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.174604 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.174746 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:21.175062 6232 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.175226 13537 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.175274 13537 raft_consensus.cc:482] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.175315 13537 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.176043 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63] alter complete (version 3)
I20240503 07:59:21.176296 13537 raft_consensus.cc:504] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.176620 13537 leader_election.cc:288] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.177735 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8cf3d6a3ace46769854ed61f340fb25" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.177840 8575 raft_consensus.cc:3037] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
W20240503 07:59:21.178584 6235 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.178699 6235 leader_election.cc:334] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.178709 8575 raft_consensus.cc:2454] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 2.
I20240503 07:59:21.178702 6156 catalog_manager.cc:5484] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.179065 6232 leader_election.cc:302] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.179255 13537 raft_consensus.cc:2781] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.179368 13537 raft_consensus.cc:686] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [term 2 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:21.179497 13537 consensus_queue.cc:237] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.181833 6156 catalog_manager.cc:5484] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 07:59:21.187541 13999 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.187582 11524 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.187618 13999 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.187673 11524 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.187942 13999 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.188004 11524 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.188196 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.188297 8975 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.188658 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:21.188802 6559 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:21.188824 8516 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
W20240503 07:59:21.188954 8517 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.189067 8910 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:21.189123 13999 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.189167 13999 raft_consensus.cc:482] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
W20240503 07:59:21.189155 8913 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.189200 13999 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.189306 11524 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 07:59:21.189353 11524 raft_consensus.cc:482] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9553fe3c4ccd4e91b7c176aad82fec09)
I20240503 07:59:21.189378 11524 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.189872 13999 raft_consensus.cc:504] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.190069 11524 raft_consensus.cc:504] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.190410 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00bcbd5440ff4d56b1975edc4041b428" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.190495 11524 leader_election.cc:288] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:21.190506 8975 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.190860 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a833fb4980c04673a335d020a3ad3dd3" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:21.190968 6559 raft_consensus.cc:3037] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:21.190999 13999 leader_election.cc:288] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
W20240503 07:59:21.191109 8517 leader_election.cc:334] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.191367 8975 raft_consensus.cc:2454] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 2.
I20240503 07:59:21.191607 8516 leader_election.cc:302] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.191684 6559 raft_consensus.cc:2454] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 2.
I20240503 07:59:21.191752 13999 raft_consensus.cc:2781] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:21.191825 13999 raft_consensus.cc:686] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.191975 13999 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } }
I20240503 07:59:21.192214 8910 leader_election.cc:302] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:21.192364 11524 raft_consensus.cc:2781] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 07:59:21.192443 8913 leader_election.cc:334] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.192463 11524 raft_consensus.cc:686] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:21.192570 11524 consensus_queue.cc:237] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } }
I20240503 07:59:21.194847 6156 catalog_manager.cc:5484] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.195057 6151 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 1 to 2, leader changed from 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134) to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.209089 12241 consensus_queue.cc:1035] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.211644 11508 consensus_queue.cc:1035] T 7af0459cb687453eb98aa1db18f39828 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.278317 6151 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.278470 6151 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:21.280220 6151 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.280828 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:21.280943 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:21.281117 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca739ac52346ae81c06834fbb781ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:21 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:21.281198 14033 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:21.281232 14034 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:21.281344 14033 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.281355 14035 tablet_replica.cc:330] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:21.281388 14034 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:21.281451 14033 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.281459 14035 raft_consensus.cc:2229] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:21.281553 14035 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.281607 14034 raft_consensus.cc:2258] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:21.282076 14033 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282115 14035 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282114 14034 ts_tablet_manager.cc:1875] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:21.282452 8843 raft_consensus.cc:1261] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:21.283514 11524 consensus_queue.cc:1035] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:21.284332 14034 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.284384 14033 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.284417 14034 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.284440 14033 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.284660 14034 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:21.284754 14033 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:21.285362 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
I20240503 07:59:21.285779 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
I20240503 07:59:21.285948 14035 ts_tablet_manager.cc:1888] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:21.286021 14035 log.cc:1196] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/86ca739ac52346ae81c06834fbb781ee
I20240503 07:59:21.286571 14035 ts_tablet_manager.cc:1909] T 86ca739ac52346ae81c06834fbb781ee P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:21.287268 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 86ca739ac52346ae81c06834fbb781ee (table rebalancer_dynamic_table_new_name [id=4936bfad579245c1ad66bdab2bb74f63]) successfully deleted
W20240503 07:59:21.287348 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.309939 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:21.311094 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.315935 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316205 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316327 8561 tablet_service.cc:1467] Processing CreateTablet for tablet ef76600ded194f318cf97f157845bea8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316368 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.316584 8561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef76600ded194f318cf97f157845bea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316687 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.316808 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.317026 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.317080 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.317214 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.319504 8957 tablet_service.cc:1467] Processing CreateTablet for tablet ef76600ded194f318cf97f157845bea8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.319742 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef76600ded194f318cf97f157845bea8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.321230 14039 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.322088 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 379cff86f1354d6f98951becdd34cda0 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.322207 14038 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:21.322347 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379cff86f1354d6f98951becdd34cda0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.322839 14039 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.323117 14038 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.323891 14039 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.323987 14039 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:21.324425 14039 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.324520 14039 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.324537 14042 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:21.324553 14039 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.324658 14039 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.322088 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 8c105dd8623848e9826de6a18d1c54d8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=76f1d9e057c04aafbfd8d0300087b1d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:21.324945 14039 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.324973 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c105dd8623848e9826de6a18d1c54d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.325107 14039 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.325623 14042 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.325877 14039 tablet_bootstrap.cc:654] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.326552 14042 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:21.326627 14042 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:21.326956 14039 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.327029 14039 ts_tablet_manager.cc:1367] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.327031 14042 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.327103 14042 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.327128 14042 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.327234 14042 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.327440 14039 raft_consensus.cc:348] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.327529 14039 raft_consensus.cc:374] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.327580 14039 raft_consensus.cc:729] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.327739 14039 consensus_queue.cc:260] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.328016 14039 ts_tablet_manager.cc:1398] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.328554 14038 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:21.328645 14038 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:21.329097 14038 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.329205 14038 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.329281 14038 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.329432 14038 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.329706 14038 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.329897 14042 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:21.330044 14042 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:21.330840 14042 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.331823 14042 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:21.331918 14042 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:21.332355 14042 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.332475 14042 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.332517 14042 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.332631 14042 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.332926 14042 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.333130 14037 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.334491 14037 tablet_bootstrap.cc:654] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.337101 14037 tablet_bootstrap.cc:492] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.337368 14037 ts_tablet_manager.cc:1367] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:21.337751 14037 raft_consensus.cc:348] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.337837 14037 raft_consensus.cc:374] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.337867 14037 raft_consensus.cc:729] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.337980 14037 consensus_queue.cc:260] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.340926 14037 ts_tablet_manager.cc:1398] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:21.341084 14037 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.341756 14037 tablet_bootstrap.cc:654] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.342875 14037 tablet_bootstrap.cc:492] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.342937 14037 ts_tablet_manager.cc:1367] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.343247 14037 raft_consensus.cc:348] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.343323 14037 raft_consensus.cc:374] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.343343 14037 raft_consensus.cc:729] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.343458 14037 consensus_queue.cc:260] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.343685 14037 ts_tablet_manager.cc:1398] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:21.344992 14037 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.345890 14037 tablet_bootstrap.cc:654] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.346846 14037 tablet_bootstrap.cc:492] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.346920 14037 ts_tablet_manager.cc:1367] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.347294 14037 raft_consensus.cc:348] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.347371 14037 raft_consensus.cc:374] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.347399 14037 raft_consensus.cc:729] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.347501 14037 consensus_queue.cc:260] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.347945 14037 ts_tablet_manager.cc:1398] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:21.363312 11508 raft_consensus.cc:482] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.363404 11508 raft_consensus.cc:504] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.363751 11508 leader_election.cc:288] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:21.364040 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.364059 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:21.364133 8575 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:21.364169 6690 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:21.364373 8776 leader_election.cc:302] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.364538 11508 raft_consensus.cc:2781] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.364590 11508 raft_consensus.cc:482] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.364625 11508 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.365329 11508 raft_consensus.cc:504] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.365609 11508 leader_election.cc:288] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:21.365856 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.365926 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379cff86f1354d6f98951becdd34cda0" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:21.365958 8575 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.366024 6690 raft_consensus.cc:3037] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.366664 6690 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:21.366739 8575 raft_consensus.cc:2454] T 379cff86f1354d6f98951becdd34cda0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:21.366911 8778 leader_election.cc:302] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:21.367266 11508 raft_consensus.cc:2781] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.367344 11508 raft_consensus.cc:686] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:21.367458 11508 consensus_queue.cc:237] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:21.369303 6151 catalog_manager.cc:5484] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.376536 13999 raft_consensus.cc:482] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.376610 13999 raft_consensus.cc:504] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.376916 13999 leader_election.cc:288] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.377213 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef76600ded194f318cf97f157845bea8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.377344 8975 raft_consensus.cc:2454] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.377755 8516 leader_election.cc:302] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.377913 13999 raft_consensus.cc:2781] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.377961 13999 raft_consensus.cc:482] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.377995 13999 raft_consensus.cc:3037] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.377993 8517 leader_election.cc:334] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.378691 13999 raft_consensus.cc:504] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.379004 13999 leader_election.cc:288] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.379640 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef76600ded194f318cf97f157845bea8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.379750 8975 raft_consensus.cc:3037] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.379940 8517 leader_election.cc:334] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.380473 8975 raft_consensus.cc:2454] T ef76600ded194f318cf97f157845bea8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.380745 8516 leader_election.cc:302] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:21.380946 13999 raft_consensus.cc:2781] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.381037 13999 raft_consensus.cc:686] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.381153 13999 consensus_queue.cc:237] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.381598 6151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:21.383132 6154 catalog_manager.cc:5484] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.386955 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 07d78407a3354848826ab84e5b750ce0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]), partition=
I20240503 07:59:21.387185 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d78407a3354848826ab84e5b750ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.388835 14037 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:21.389144 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 07d78407a3354848826ab84e5b750ce0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]), partition=
I20240503 07:59:21.389351 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d78407a3354848826ab84e5b750ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:21.389604 14037 tablet_bootstrap.cc:654] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.390616 14037 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:21.390735 14037 ts_tablet_manager.cc:1367] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:21.391162 14039 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:21.391178 14037 raft_consensus.cc:348] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.391291 14037 raft_consensus.cc:374] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.391324 14037 raft_consensus.cc:729] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.391467 14037 consensus_queue.cc:260] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.391750 14037 ts_tablet_manager.cc:1398] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.392012 14039 tablet_bootstrap.cc:654] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:21.393143 14039 tablet_bootstrap.cc:492] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:21.393235 14039 ts_tablet_manager.cc:1367] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:21.393633 14039 raft_consensus.cc:348] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.393714 14039 raft_consensus.cc:374] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:21.393735 14039 raft_consensus.cc:729] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:21.393867 14039 consensus_queue.cc:260] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.394117 14039 ts_tablet_manager.cc:1398] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:21.517561 6298 raft_consensus.cc:1261] T f635e30fb6974fa1a79d8a46eecc3d5e P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.517930 13999 consensus_queue.cc:1035] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 07:59:21.520320 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.570469 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.582266 6559 raft_consensus.cc:1261] T f534eefb7c334889a9c5d02175ae2d94 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.583628 13922 consensus_queue.cc:1035] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.588639 13924 raft_consensus.cc:482] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.588757 13924 raft_consensus.cc:504] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.589196 13924 leader_election.cc:288] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.589437 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d78407a3354848826ab84e5b750ce0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:21.589558 8575 raft_consensus.cc:2454] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:21.589790 8910 leader_election.cc:302] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.589927 13924 raft_consensus.cc:2781] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.589993 13924 raft_consensus.cc:482] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.590036 13924 raft_consensus.cc:3037] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:21.590153 8913 leader_election.cc:334] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.590934 13924 raft_consensus.cc:504] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:21.591219 13924 leader_election.cc:288] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:21.591456 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d78407a3354848826ab84e5b750ce0" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:21.591567 8575 raft_consensus.cc:3037] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.592420 8575 raft_consensus.cc:2454] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:21.592674 8910 leader_election.cc:302] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.592869 13924 raft_consensus.cc:2781] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.592976 13924 raft_consensus.cc:686] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:21.593102 13924 consensus_queue.cc:237] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
W20240503 07:59:21.593295 8913 leader_election.cc:334] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:21.594835 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.601714 6154 catalog_manager.cc:5484] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.630599 14056 raft_consensus.cc:482] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:21.630698 14056 raft_consensus.cc:504] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.631099 14056 leader_election.cc:288] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:21.631387 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:21.631400 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:21.631490 8975 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.631510 8843 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:21.631839 8516 leader_election.cc:302] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.632083 14056 raft_consensus.cc:2781] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:21.632154 14056 raft_consensus.cc:482] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:21.632200 14056 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.633081 14056 raft_consensus.cc:504] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.633407 14056 leader_election.cc:288] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:21.633641 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:21.633730 8975 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.633714 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c105dd8623848e9826de6a18d1c54d8" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:21.633816 8843 raft_consensus.cc:3037] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:21.634433 8975 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.634621 8843 raft_consensus.cc:2454] T 8c105dd8623848e9826de6a18d1c54d8 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:21.634682 8516 leader_election.cc:302] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:21.634848 14056 raft_consensus.cc:2781] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:21.634945 14056 raft_consensus.cc:686] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:21.635144 14056 consensus_queue.cc:237] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:21.637208 6154 catalog_manager.cc:5484] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:21.706758 8843 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 07:59:21.707135 13989 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 07:59:21.709969 6365 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b61209b73f1f4872b5eccc823b1fa3df. This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.711203 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.711352 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:21.715363 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.719358 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:21.720635 8975 raft_consensus.cc:1261] T 6f260c4c4e7e4fcc8fbce11064bdc338 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 07:59:21.721194 13537 consensus_queue.cc:1035] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 07:59:21.723016 8575 raft_consensus.cc:1261] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:21.723534 12930 consensus_queue.cc:1035] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:21.723927 8913 consensus_peers.cc:485] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.725504 8575 raft_consensus.cc:1261] T c8cf3d6a3ace46769854ed61f340fb25 P c8d71787690245e681c85a087086e7ed [term 2 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.726090 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.727475 6151 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.727980 14063 consensus_queue.cc:1035] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.728140 6559 raft_consensus.cc:1261] T a833fb4980c04673a335d020a3ad3dd3 P ad910f92ef9d4db8a8962cecba289f8a [term 2 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:21.728307 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:21.728596 12930 consensus_queue.cc:1035] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:21.728680 14065 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.730031 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4]). Expected version 1 got 0
I20240503 07:59:21.732908 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 07:59:21.733187 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.733196 14062 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.737560 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 1)
W20240503 07:59:21.740212 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:21.774189 6298 raft_consensus.cc:3037] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 1 LEARNER]: Advancing to term 2
I20240503 07:59:21.775300 6298 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:21.775647 14056 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: NON_VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.777141 14056 raft_consensus.cc:1050] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
W20240503 07:59:21.777212 14056 raft_consensus.cc:1053] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: Unable to promote non-voter c167a0a433d34b74affed13cbf96e302: Illegal state: Leader has not yet committed an operation in its own term
I20240503 07:59:21.781292 8975 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 07:59:21.781610 14056 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 07:59:21.784348 14056 raft_consensus.cc:1050] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed: attempting to promote NON_VOTER c167a0a433d34b74affed13cbf96e302 to VOTER
I20240503 07:59:21.784726 14056 consensus_queue.cc:237] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } }
I20240503 07:59:21.786801 6298 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 LEARNER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 07:59:21.786800 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.787143 8975 raft_consensus.cc:1261] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 07:59:21.787261 13922 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:21.787760 14070 consensus_queue.cc:1035] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:21.789089 13999 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.789633 6298 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P c167a0a433d34b74affed13cbf96e302 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.791306 8975 raft_consensus.cc:2932] T 00bcbd5440ff4d56b1975edc4041b428 P 3d08a06d53b14381bee6223c4e1a6027 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } } }
I20240503 07:59:21.792114 6151 catalog_manager.cc:5484] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed reported cstate change: config changed from index 2 to 4, c167a0a433d34b74affed13cbf96e302 (127.0.99.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 07:59:21.833552 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:21.840432 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:21.844887 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.845268 14062 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:21.845544 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:21.845858 14065 tablet.cc:1620] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:21.849850 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 2)
I20240503 07:59:21.868474 13953 consensus_queue.cc:1035] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.871040 12241 consensus_queue.cc:1035] T 379cff86f1354d6f98951becdd34cda0 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:21.917565 14056 consensus_queue.cc:1035] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:21.944645 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:21.952306 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:21.957231 14065 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.957047 14062 tablet.cc:1722] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:21.961932 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4] alter complete (version 3)
W20240503 07:59:21.968989 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): CreateTablet RPC failed for tablet ef76600ded194f318cf97f157845bea8: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.047253 14056 consensus_queue.cc:1035] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.050745 13922 consensus_queue.cc:1035] T 8c105dd8623848e9826de6a18d1c54d8 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.064800 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.064925 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.066797 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.067298 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d78407a3354848826ab84e5b750ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:22.067605 14078 tablet_replica.cc:330] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:22.067606 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d78407a3354848826ab84e5b750ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:22.067775 14078 raft_consensus.cc:2229] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:22.067885 14079 tablet_replica.cc:330] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:22.067996 14078 raft_consensus.cc:2258] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.068022 14079 raft_consensus.cc:2229] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.068149 14079 raft_consensus.cc:2258] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.068523 14078 ts_tablet_manager.cc:1875] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.068671 14079 ts_tablet_manager.cc:1875] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.070245 14078 ts_tablet_manager.cc:1888] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.070364 14078 log.cc:1196] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.070679 14078 ts_tablet_manager.cc:1909] T 07d78407a3354848826ab84e5b750ce0 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:22.070676 14079 ts_tablet_manager.cc:1888] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.070744 14079 log.cc:1196] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/07d78407a3354848826ab84e5b750ce0
I20240503 07:59:22.071005 14079 ts_tablet_manager.cc:1909] T 07d78407a3354848826ab84e5b750ce0 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:22.071456 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4]) successfully deleted
I20240503 07:59:22.071673 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 07d78407a3354848826ab84e5b750ce0 (table rebalancer_dynamic_table_new_name [id=bbf0d995a3ac4e5ea749ee189def71a4]) successfully deleted
I20240503 07:59:22.122449 14080 ts_tablet_manager.cc:902] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Initiating tablet copy from peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:22.122948 14080 tablet_copy_client.cc:308] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:41651
I20240503 07:59:22.123373 6450 tablet_copy_service.cc:140] P d1f36d57661b4429a4be8d10b58d8ff7: Received BeginTabletCopySession request for tablet b61209b73f1f4872b5eccc823b1fa3df from peer a97073f89fd44702bccb74a9b76f858d ({username='slave'} at 127.0.99.132:38973)
I20240503 07:59:22.123466 6450 tablet_copy_service.cc:161] P d1f36d57661b4429a4be8d10b58d8ff7: Beginning new tablet copy session on tablet b61209b73f1f4872b5eccc823b1fa3df from peer a97073f89fd44702bccb74a9b76f858d at {username='slave'} at 127.0.99.132:38973: session id = a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df
I20240503 07:59:22.124198 6450 tablet_copy_source_session.cc:205] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 07:59:22.125017 14080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b61209b73f1f4872b5eccc823b1fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.126794 14080 tablet_copy_client.cc:783] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 0 data blocks...
I20240503 07:59:22.126932 14080 tablet_copy_client.cc:647] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Starting download of 1 WAL segments...
I20240503 07:59:22.128077 14080 tablet_copy_client.cc:523] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 07:59:22.129439 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:22.134068 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 07:59:22.134274 14080 tablet_bootstrap.cc:492] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Bootstrap complete.
I20240503 07:59:22.134434 14080 ts_tablet_manager.cc:1367] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:22.134845 14080 raft_consensus.cc:348] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:22.134964 14080 raft_consensus.cc:729] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: LEARNER
I20240503 07:59:22.135092 14080 consensus_queue.cc:260] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: NON_VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: true } }
I20240503 07:59:22.135437 14080 ts_tablet_manager.cc:1398] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:22.135808 6450 tablet_copy_service.cc:342] P d1f36d57661b4429a4be8d10b58d8ff7: Request end of tablet copy session a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df received from {username='slave'} at 127.0.99.132:38973
I20240503 07:59:22.135877 6450 tablet_copy_service.cc:433] P d1f36d57661b4429a4be8d10b58d8ff7: ending tablet copy session a97073f89fd44702bccb74a9b76f858d-b61209b73f1f4872b5eccc823b1fa3df on tablet b61209b73f1f4872b5eccc823b1fa3df with peer a97073f89fd44702bccb74a9b76f858d
I20240503 07:59:22.168301 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:22.169857 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.173269 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.173480 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.173539 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.173717 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.175870 14086 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.176039 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 2e77ed2536c24373b141868366d836ad (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]), partition=
I20240503 07:59:22.176291 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e77ed2536c24373b141868366d836ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.176451 14087 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.176851 14086 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.177420 14087 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.178457 14090 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.178481 14086 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.178568 14086 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:22.179028 14086 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.179152 14086 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.179196 14086 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.179339 14086 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.179419 14090 tablet_bootstrap.cc:654] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.179647 14086 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.180301 14087 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.180428 14087 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:22.180757 14090 tablet_bootstrap.cc:492] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.180864 14090 ts_tablet_manager.cc:1367] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.180752 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:22.181259 14090 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.181360 14090 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.181385 14090 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.181483 14090 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.181718 14090 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.183933 14087 raft_consensus.cc:348] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.184473 14087 raft_consensus.cc:374] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.184695 14087 raft_consensus.cc:729] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.185016 14087 consensus_queue.cc:260] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.186426 14087 ts_tablet_manager.cc:1398] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:22.188561 8957 tablet_service.cc:1467] Processing CreateTablet for tablet d8c681e92f9748f2b6b20958a8efed36 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189078 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c681e92f9748f2b6b20958a8efed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.189323 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6317b68edd7c4512bb96248a9cac9399 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189488 8956 tablet_service.cc:1467] Processing CreateTablet for tablet cd978026a0aa42758f07a701131fdc27 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.189545 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6317b68edd7c4512bb96248a9cac9399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.189723 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd978026a0aa42758f07a701131fdc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.191000 14087 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.191434 6539 tablet_service.cc:1467] Processing CreateTablet for tablet d8c681e92f9748f2b6b20958a8efed36 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.191707 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c681e92f9748f2b6b20958a8efed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.192171 8559 tablet_service.cc:1467] Processing CreateTablet for tablet cd978026a0aa42758f07a701131fdc27 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.192358 14087 tablet_bootstrap.cc:654] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.192430 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd978026a0aa42758f07a701131fdc27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.193202 14087 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.193253 14087 ts_tablet_manager.cc:1367] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:22.193571 14087 raft_consensus.cc:348] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.193639 14087 raft_consensus.cc:374] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.193657 14087 raft_consensus.cc:729] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.193781 14087 consensus_queue.cc:260] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.193995 14087 ts_tablet_manager.cc:1398] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:22.194133 14087 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:22.194224 14086 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.194307 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 6317b68edd7c4512bb96248a9cac9399 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=8ef05743714c43a5918244bbc3325bc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:22.194569 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6317b68edd7c4512bb96248a9cac9399. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.194924 14087 tablet_bootstrap.cc:654] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.195135 14086 tablet_bootstrap.cc:654] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.195735 14090 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.196113 14086 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.196167 14086 ts_tablet_manager.cc:1367] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:22.196478 14086 raft_consensus.cc:348] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.196543 14086 raft_consensus.cc:374] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.196569 14086 raft_consensus.cc:729] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.196655 14086 consensus_queue.cc:260] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.196702 14090 tablet_bootstrap.cc:654] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.196921 14096 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:22.196985 14086 ts_tablet_manager.cc:1398] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.197717 14096 tablet_bootstrap.cc:654] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.198264 14090 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.198350 14090 ts_tablet_manager.cc:1367] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:22.198802 14090 raft_consensus.cc:348] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.198906 14090 raft_consensus.cc:374] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.198940 14090 raft_consensus.cc:729] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.199061 14090 consensus_queue.cc:260] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.199151 14096 tablet_bootstrap.cc:492] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:22.199344 14090 ts_tablet_manager.cc:1398] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.199218 14096 ts_tablet_manager.cc:1367] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.200155 14087 tablet_bootstrap.cc:492] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:22.200235 14087 ts_tablet_manager.cc:1367] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:22.200642 14087 raft_consensus.cc:348] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.200738 14087 raft_consensus.cc:374] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.200795 14087 raft_consensus.cc:729] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.200915 14087 consensus_queue.cc:260] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.201110 14096 raft_consensus.cc:348] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.201176 14087 ts_tablet_manager.cc:1398] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.201217 14096 raft_consensus.cc:374] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.201248 14096 raft_consensus.cc:729] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.201668 14093 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:22.201346 14096 consensus_queue.cc:260] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.203207 14096 ts_tablet_manager.cc:1398] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:22.203512 14093 tablet_bootstrap.cc:654] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.205183 14093 tablet_bootstrap.cc:492] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:22.205296 14093 ts_tablet_manager.cc:1367] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:22.205695 14093 raft_consensus.cc:348] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.205791 14093 raft_consensus.cc:374] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.205828 14093 raft_consensus.cc:729] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.205955 14093 consensus_queue.cc:260] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.206243 14093 ts_tablet_manager.cc:1398] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.219100 12930 raft_consensus.cc:482] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.219172 12930 raft_consensus.cc:504] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.219487 12930 leader_election.cc:288] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.219748 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:22.219767 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:22.219843 8843 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.219869 8581 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.220104 8912 leader_election.cc:302] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:22.220257 12930 raft_consensus.cc:2781] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.220306 12930 raft_consensus.cc:482] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.220340 12930 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.221007 12930 raft_consensus.cc:504] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.221298 12930 leader_election.cc:288] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.221590 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:22.221696 8843 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.221771 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e77ed2536c24373b141868366d836ad" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:22.221887 8581 raft_consensus.cc:3037] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.222442 8843 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.222733 8912 leader_election.cc:302] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:22.222865 8581 raft_consensus.cc:2454] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.222903 12930 raft_consensus.cc:2781] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.222972 12930 raft_consensus.cc:686] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:22.223083 12930 consensus_queue.cc:237] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.223371 13922 raft_consensus.cc:482] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.223453 13922 raft_consensus.cc:504] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.224298 13922 leader_election.cc:288] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:22.224668 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd978026a0aa42758f07a701131fdc27" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:22.224810 8975 raft_consensus.cc:2454] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:22.225113 8516 leader_election.cc:302] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
W20240503 07:59:22.225193 8517 leader_election.cc:334] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.225100 6151 catalog_manager.cc:5484] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:22.225247 13922 raft_consensus.cc:2781] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.225304 13922 raft_consensus.cc:482] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.225355 13922 raft_consensus.cc:3037] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.226104 13922 raft_consensus.cc:504] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.226380 13922 leader_election.cc:288] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:22.226680 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd978026a0aa42758f07a701131fdc27" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:22.226771 8975 raft_consensus.cc:3037] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.227499 8517 leader_election.cc:334] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.227643 8975 raft_consensus.cc:2454] T cd978026a0aa42758f07a701131fdc27 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:22.227908 8516 leader_election.cc:302] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.228294 13922 raft_consensus.cc:2781] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.228385 13922 raft_consensus.cc:686] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:22.228487 13922 consensus_queue.cc:237] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:22.230325 6151 catalog_manager.cc:5484] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:22.246703 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.259523 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.262571 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:22.312248 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.315135 12930 raft_consensus.cc:482] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.315225 12930 raft_consensus.cc:504] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.315567 12930 leader_election.cc:288] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:22.315860 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c681e92f9748f2b6b20958a8efed36" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:22.315996 6559 raft_consensus.cc:2454] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:22.316285 8910 leader_election.cc:302] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:22.316459 12930 raft_consensus.cc:2781] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.316519 12930 raft_consensus.cc:482] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.316560 12930 raft_consensus.cc:3037] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.316584 8913 leader_election.cc:334] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.317337 12930 raft_consensus.cc:504] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.317607 12930 leader_election.cc:288] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925)
I20240503 07:59:22.317870 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c681e92f9748f2b6b20958a8efed36" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:22.317970 6559 raft_consensus.cc:3037] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.318491 8913 leader_election.cc:334] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.318809 6559 raft_consensus.cc:2454] T d8c681e92f9748f2b6b20958a8efed36 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:22.319053 8910 leader_election.cc:302] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.319211 12930 raft_consensus.cc:2781] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.319298 12930 raft_consensus.cc:686] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:22.319437 12930 consensus_queue.cc:237] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } }
I20240503 07:59:22.321723 6156 catalog_manager.cc:5484] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.334833 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:22.337989 8581 raft_consensus.cc:1261] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.338047 8843 raft_consensus.cc:1261] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:22.338348 12930 consensus_queue.cc:1035] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.338482 11524 consensus_queue.cc:1035] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:22.340962 6156 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
I20240503 07:59:22.341604 6151 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
I20240503 07:59:22.341991 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.342268 14062 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:22.343001 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.343252 14065 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:22.344327 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:22.345428 14107 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.345839 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 1)
I20240503 07:59:22.348187 6151 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d]). Expected version 1 got 0
W20240503 07:59:22.361014 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.373528 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.450675 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:22.455861 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.456004 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.456187 14062 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:22.456323 14107 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:22.457273 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:22.457569 14065 tablet.cc:1620] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
W20240503 07:59:22.459273 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.461278 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 2)
W20240503 07:59:22.491883 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.498047 6690 raft_consensus.cc:1203] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Deduplicated request from leader. Original: 1.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 07:59:22.508476 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.544335 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.562091 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:22.567549 14062 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.568521 14065 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:22.569221 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.572530 14107 tablet.cc:1722] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:22.573702 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d] alter complete (version 3)
I20240503 07:59:22.594962 14108 raft_consensus.cc:482] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.595068 14108 raft_consensus.cc:504] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.595415 14108 leader_election.cc:288] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:22.595978 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6317b68edd7c4512bb96248a9cac9399" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:22.596104 6298 raft_consensus.cc:2454] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:22.596403 8776 leader_election.cc:302] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:22.596571 14108 raft_consensus.cc:2781] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 07:59:22.596596 8779 leader_election.cc:334] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.596632 14108 raft_consensus.cc:482] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.596688 14108 raft_consensus.cc:3037] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.597697 14108 raft_consensus.cc:504] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.598016 14108 leader_election.cc:288] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:22.598287 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6317b68edd7c4512bb96248a9cac9399" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:22.598397 6298 raft_consensus.cc:3037] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:22.598940 8779 leader_election.cc:334] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.599237 6298 raft_consensus.cc:2454] T 6317b68edd7c4512bb96248a9cac9399 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:22.599494 8776 leader_election.cc:302] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters: 9553fe3c4ccd4e91b7c176aad82fec09
I20240503 07:59:22.599642 14108 raft_consensus.cc:2781] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.599711 14108 raft_consensus.cc:686] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:22.599864 14108 consensus_queue.cc:237] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:22.601778 6154 catalog_manager.cc:5484] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:22.627704 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.629621 14070 consensus_queue.cc:1035] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:22.633198 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:22.638283 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.677311 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.677446 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:22.679280 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.679746 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:22.679832 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:22.680058 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e77ed2536c24373b141868366d836ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:22 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:22.680159 14112 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:22.680248 14113 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:22.680346 14112 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.680361 14114 tablet_replica.cc:330] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:22.680410 14113 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:22.680477 14114 raft_consensus.cc:2229] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:22.680483 14112 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.680536 14113 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.680656 14114 raft_consensus.cc:2258] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:22.681011 14112 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.681188 14113 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.681191 14114 ts_tablet_manager.cc:1875] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:22.683362 14112 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683393 14114 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683393 14113 ts_tablet_manager.cc:1888] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:22.683454 14113 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683454 14114 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683477 14112 log.cc:1196] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2e77ed2536c24373b141868366d836ad
I20240503 07:59:22.683727 14113 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:22.683727 14114 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:22.683751 14112 ts_tablet_manager.cc:1909] T 2e77ed2536c24373b141868366d836ad P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:22.684304 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
I20240503 07:59:22.684459 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
I20240503 07:59:22.684469 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 2e77ed2536c24373b141868366d836ad (table rebalancer_dynamic_table_new_name [id=3414eb4f219c48749e88f6c6e6adb97d]) successfully deleted
W20240503 07:59:22.731793 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.750429 14106 consensus_queue.cc:1035] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:22.754549 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.781143 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:22.781433 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:22.786182 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786188 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786392 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]), partition=
I20240503 07:59:22.786443 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.786458 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.786593 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0454f40ebacd4a4bb2854a7bcf765295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:22.788123 14117 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:22.788870 14119 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:22.788924 14118 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:22.789103 14117 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.789631 14119 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.789988 14118 tablet_bootstrap.cc:654] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:22.790270 14117 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:22.790376 14117 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:22.790653 14119 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:22.790731 14119 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:22.790866 14117 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.790982 14117 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.791018 14117 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.791132 14117 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791204 14119 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791285 14119 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.791312 14119 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.791407 14117 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:22.791425 14119 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.791663 14119 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:22.793313 14118 tablet_bootstrap.cc:492] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:22.793519 14118 ts_tablet_manager.cc:1367] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:22.797345 14118 raft_consensus.cc:348] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.797463 14118 raft_consensus.cc:374] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:22.797508 14118 raft_consensus.cc:729] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:22.797632 14118 consensus_queue.cc:260] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.797910 14118 ts_tablet_manager.cc:1398] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240503 07:59:22.839605 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:22.867251 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:22.911473 13018 raft_consensus.cc:482] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:22.911594 13018 raft_consensus.cc:504] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.911927 13018 leader_election.cc:288] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.912199 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:22.912243 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:22.912362 8843 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:22.912372 8581 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:22.912621 6494 leader_election.cc:302] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:22.912801 13018 raft_consensus.cc:2781] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:22.912865 13018 raft_consensus.cc:482] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:22.912914 13018 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.913751 13018 raft_consensus.cc:504] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.914072 13018 leader_election.cc:288] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:22.914427 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:22.914544 8843 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.914861 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0454f40ebacd4a4bb2854a7bcf765295" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:22.914958 8581 raft_consensus.cc:3037] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:22.915359 8843 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:22.915637 6496 leader_election.cc:302] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:22.915740 8581 raft_consensus.cc:2454] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:22.915803 13018 raft_consensus.cc:2781] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:22.915896 13018 raft_consensus.cc:686] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:22.916023 13018 consensus_queue.cc:237] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:22.918051 6154 catalog_manager.cc:5484] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:22.956140 14109 raft_consensus.cc:1050] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7: attempting to promote NON_VOTER a97073f89fd44702bccb74a9b76f858d to VOTER
I20240503 07:59:22.956606 14109 consensus_queue.cc:237] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } }
I20240503 07:59:22.958097 6690 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 LEARNER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:22.958139 8843 raft_consensus.cc:1261] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 07:59:22.958770 12434 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:22.959033 12434 consensus_queue.cc:1035] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:22.960264 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:22.961519 12434 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.963371 8843 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P 616fdda6ba714cb7be01a564de2b9db2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.963627 6690 raft_consensus.cc:2932] T b61209b73f1f4872b5eccc823b1fa3df P a97073f89fd44702bccb74a9b76f858d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } } }
I20240503 07:59:22.966161 6154 catalog_manager.cc:5484] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: config changed from index 5 to 7, a97073f89fd44702bccb74a9b76f858d (127.0.99.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553fe3c4ccd4e91b7c176aad82fec09" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:22.983368 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 551f25dcee034b7894498e3c748b0f0e: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.006405 12241 consensus_queue.cc:1035] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.009367 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:23.037837 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.041615 8581 raft_consensus.cc:1261] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.041718 8843 raft_consensus.cc:1261] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.042023 13018 consensus_queue.cc:1035] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.042301 13018 consensus_queue.cc:1035] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.045094 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.045352 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.045655 14107 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.046299 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.046572 14065 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:23.048272 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.049297 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1]). Expected version 1 got 0
I20240503 07:59:23.053551 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 1)
I20240503 07:59:23.053885 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.054746 14137 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.147296 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:23.153956 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154016 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154248 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.154284 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.154399 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.154628 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157052 14138 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.157387 6151 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:23.157423 8559 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.157557 6277 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.157678 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157774 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.157811 14139 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:23.158317 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.158538 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.158701 14139 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.159637 14118 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.159776 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 05c9f3f3dc934c828afdb98aefbd628e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.159976 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c9f3f3dc934c828afdb98aefbd628e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.159922 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8636cb24f7435ea195b349304e40d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.160156 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8636cb24f7435ea195b349304e40d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.160514 14118 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.160631 8824 tablet_service.cc:1467] Processing CreateTablet for tablet f7de7be7d836442393dbd9f9f63a3cab (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6fa64ec7ca394db6b9deada56a8e3bfd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.160902 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7de7be7d836442393dbd9f9f63a3cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.161864 14118 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.161953 14118 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.162249 14119 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:23.162340 14118 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.162431 14118 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.162458 14118 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.162586 14118 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.162849 14118 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.163009 14118 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.163043 14119 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.163803 14118 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.163956 14117 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:23.164856 14117 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.165864 14138 tablet_bootstrap.cc:654] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.167317 14119 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:23.167385 14119 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:23.167742 14119 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.167826 14119 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.167853 14119 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.167968 14119 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.168195 14142 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.168215 14119 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.169176 14142 tablet_bootstrap.cc:654] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.169652 14138 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.169773 14138 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:23.169800 14140 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.170226 14138 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.170389 14138 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.170521 14138 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.170701 14140 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.170709 14138 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.171747 14138 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.171984 14118 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.172078 14118 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:23.172484 14118 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.172575 14118 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.172603 14118 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.172693 14118 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.172976 14118 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.173900 14139 tablet_bootstrap.cc:492] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:23.173986 14139 ts_tablet_manager.cc:1367] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:23.174392 14139 raft_consensus.cc:348] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.174491 14139 raft_consensus.cc:374] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.174522 14139 raft_consensus.cc:729] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.174643 14139 consensus_queue.cc:260] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.174901 14139 ts_tablet_manager.cc:1398] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.176604 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.176954 14137 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:23.177275 14142 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.177349 14142 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:23.177542 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.177785 14107 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.177752 14142 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.177843 14142 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.177874 14142 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.177980 14142 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.178231 14142 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.178411 14142 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.178553 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:23.178777 14065 tablet.cc:1620] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:23.178828 14117 tablet_bootstrap.cc:492] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:23.178891 14117 ts_tablet_manager.cc:1367] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20240503 07:59:23.179304 14117 raft_consensus.cc:348] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.179404 14117 raft_consensus.cc:374] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.179441 14117 raft_consensus.cc:729] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.179546 14117 consensus_queue.cc:260] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.179782 14117 ts_tablet_manager.cc:1398] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.181383 14142 tablet_bootstrap.cc:654] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.182521 14140 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.182636 14140 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:23.183087 14140 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.183231 14140 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.183296 14140 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.183452 14140 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.183789 14140 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.184818 14142 tablet_bootstrap.cc:492] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.184942 6151 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 2)
I20240503 07:59:23.185173 14142 ts_tablet_manager.cc:1367] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:23.185603 14142 raft_consensus.cc:348] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.185760 14142 raft_consensus.cc:374] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.185830 14142 raft_consensus.cc:729] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.185978 14142 consensus_queue.cc:260] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.186542 14142 ts_tablet_manager.cc:1398] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.191937 14146 raft_consensus.cc:482] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.192009 14146 raft_consensus.cc:504] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.192354 14146 leader_election.cc:288] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:23.192569 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:23.192632 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:23.192682 8843 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.192739 6298 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.192970 8516 leader_election.cc:302] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.193353 14146 raft_consensus.cc:2781] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.193414 14146 raft_consensus.cc:482] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.193459 14146 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.194120 14146 raft_consensus.cc:504] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.194401 14146 leader_election.cc:288] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:23.194675 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:23.194775 6298 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.194814 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7de7be7d836442393dbd9f9f63a3cab" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:23.194891 8843 raft_consensus.cc:3037] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.195627 6298 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.195627 8843 raft_consensus.cc:2454] T f7de7be7d836442393dbd9f9f63a3cab P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.195915 8516 leader_election.cc:302] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.196051 14146 raft_consensus.cc:2781] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.196131 14146 raft_consensus.cc:686] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.196242 14146 consensus_queue.cc:237] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.197997 6155 catalog_manager.cc:5484] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:23.205488 14063 raft_consensus.cc:482] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.205569 14063 raft_consensus.cc:504] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.205947 14063 leader_election.cc:288] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:23.206161 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:23.206214 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:23.206280 6559 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.206313 6430 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.206503 6232 leader_election.cc:302] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.206650 14063 raft_consensus.cc:2781] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.206705 14063 raft_consensus.cc:482] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.206749 14063 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.207533 14063 raft_consensus.cc:504] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.207850 14063 leader_election.cc:288] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:23.208093 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:23.208187 6559 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.208250 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8636cb24f7435ea195b349304e40d8" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:23.208323 6430 raft_consensus.cc:3037] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.208809 6559 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.209012 6430 raft_consensus.cc:2454] T 5a8636cb24f7435ea195b349304e40d8 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.209077 6232 leader_election.cc:302] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.209226 14063 raft_consensus.cc:2781] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.209297 14063 raft_consensus.cc:686] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:23.209411 14063 consensus_queue.cc:237] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.210974 6155 catalog_manager.cc:5484] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.255878 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.262044 14146 raft_consensus.cc:482] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.262133 14146 raft_consensus.cc:504] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.262485 14146 leader_election.cc:288] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:23.262737 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.262784 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:23.262866 8976 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.262905 6691 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.263162 8516 leader_election.cc:302] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.263331 14146 raft_consensus.cc:2781] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.263389 14146 raft_consensus.cc:482] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.263432 14146 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.264115 14146 raft_consensus.cc:504] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.264393 14146 leader_election.cc:288] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:23.264657 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.264699 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c9f3f3dc934c828afdb98aefbd628e" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:23.264747 8976 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.264813 6691 raft_consensus.cc:3037] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.265497 8976 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.265610 6691 raft_consensus.cc:2454] T 05c9f3f3dc934c828afdb98aefbd628e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.265745 8516 leader_election.cc:302] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.266017 14146 raft_consensus.cc:2781] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.266120 14146 raft_consensus.cc:686] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.266486 14146 consensus_queue.cc:237] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.268529 6155 catalog_manager.cc:5484] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:23.287719 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:23.292295 14137 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.292919 14107 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.296203 14065 tablet.cc:1722] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:23.297508 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1] alter complete (version 3)
W20240503 07:59:23.359516 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.399442 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.399592 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:23.401573 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.402029 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:23.402151 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:23.402402 14156 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:23.402452 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0454f40ebacd4a4bb2854a7bcf765295 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:23 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:23.402558 14157 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:23.402570 14156 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:23.402696 14157 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.402719 14158 tablet_replica.cc:330] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:23.402772 14156 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.402824 14157 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.402825 14158 raft_consensus.cc:2229] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:23.402940 14158 raft_consensus.cc:2258] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:23.403419 14157 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.403465 14156 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.403525 14158 ts_tablet_manager.cc:1875] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:23.405484 14156 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405560 14156 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405587 14157 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405587 14158 ts_tablet_manager.cc:1888] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:23.405658 14158 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405690 14157 log.cc:1196] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/0454f40ebacd4a4bb2854a7bcf765295
I20240503 07:59:23.405906 14156 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:23.405942 14158 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:23.405942 14157 ts_tablet_manager.cc:1909] T 0454f40ebacd4a4bb2854a7bcf765295 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:23.406617 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.406734 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.406872 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 0454f40ebacd4a4bb2854a7bcf765295 (table rebalancer_dynamic_table_new_name [id=67eae43efc1943c9936c9c64e936d7c1]) successfully deleted
I20240503 07:59:23.503042 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:23.508054 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.508037 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.508294 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.508299 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.510016 14117 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:23.510066 14138 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.510180 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 01ca18d3d275477f832ccd94b1958924 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]), partition=
I20240503 07:59:23.510378 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ca18d3d275477f832ccd94b1958924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.510833 14117 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.510862 14138 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.512027 14117 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:23.512105 14117 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:23.512151 14138 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.512223 14138 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.512363 14118 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.512518 14117 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512610 14117 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.512640 14117 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.512583 14138 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512676 14138 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.512696 14138 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.512756 14117 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.512951 14138 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.513053 14117 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.513228 14118 tablet_bootstrap.cc:654] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.513264 14138 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.514600 14118 tablet_bootstrap.cc:492] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.514676 14118 ts_tablet_manager.cc:1367] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:23.515009 14118 raft_consensus.cc:348] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.515093 14118 raft_consensus.cc:374] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.515122 14118 raft_consensus.cc:729] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.515219 14118 consensus_queue.cc:260] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.515460 14118 ts_tablet_manager.cc:1398] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:23.590819 14124 consensus_queue.cc:1035] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.593855 14124 consensus_queue.cc:1035] T 5a8636cb24f7435ea195b349304e40d8 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.616364 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.625484 14146 consensus_queue.cc:1035] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.629092 14070 consensus_queue.cc:1035] T f7de7be7d836442393dbd9f9f63a3cab P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:23.654918 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.668956 14146 consensus_queue.cc:1035] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.671737 14170 consensus_queue.cc:1035] T 05c9f3f3dc934c828afdb98aefbd628e P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 07:59:23.740381 14170 raft_consensus.cc:482] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.740481 14170 raft_consensus.cc:504] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.740803 14170 leader_election.cc:288] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.741061 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:23.741147 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.741194 8843 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.741284 8975 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:23.741573 8516 leader_election.cc:302] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.741833 14170 raft_consensus.cc:2781] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.741890 14170 raft_consensus.cc:482] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.741916 14170 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.742870 14170 raft_consensus.cc:504] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.743153 14170 leader_election.cc:288] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.743422 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:23.743530 8843 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.743513 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ca18d3d275477f832ccd94b1958924" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.743619 8975 raft_consensus.cc:3037] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.744266 8843 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.744422 8975 raft_consensus.cc:2454] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:23.744503 8516 leader_election.cc:302] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.744647 14170 raft_consensus.cc:2781] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.744735 14170 raft_consensus.cc:686] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:23.744881 14170 consensus_queue.cc:237] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:23.747112 6154 catalog_manager.cc:5484] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.753748 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:23.779624 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:23.785744 8560 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.785768 8956 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786029 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786051 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786434 8957 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786478 8955 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.786636 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.786756 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.788132 14118 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.788729 14138 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.789206 14118 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.789577 14138 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.789763 6408 tablet_service.cc:1467] Processing CreateTablet for tablet a4b43f6d98d9429390c74c722bc1d4d0 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.789960 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b43f6d98d9429390c74c722bc1d4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.789909 6410 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.790124 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.790604 14138 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.790669 14138 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.791074 14138 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.791169 14138 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.791203 14138 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.791316 14138 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.791577 14138 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.791761 14138 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.792454 14118 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.792533 14118 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:23.792570 14138 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.792990 14118 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.793085 14118 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.793112 14118 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.793211 14118 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.793212 6539 tablet_service.cc:1467] Processing CreateTablet for tablet ccc4f65253d34507ad704a3d88f0cfbb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.793449 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc4f65253d34507ad704a3d88f0cfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.793488 14118 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.793489 6279 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.793715 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.795363 14180 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:23.795452 14181 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:23.796252 14180 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.796960 14138 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.797021 14181 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.797046 14138 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:23.797472 14138 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.797555 14138 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.797582 14138 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.797696 14138 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.797953 14138 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.798076 14178 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.798282 14138 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:23.798415 8560 tablet_service.cc:1467] Processing CreateTablet for tablet c56117fc1ae74473b3836584a22b6b16 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ea9ea91acb2e445ca985cdce782147d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:23.798619 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56117fc1ae74473b3836584a22b6b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:23.799047 14138 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.799418 14178 tablet_bootstrap.cc:654] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.800001 14138 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:23.800074 14138 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:23.800271 14118 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:23.800866 14138 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.801029 14138 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.801075 14118 tablet_bootstrap.cc:654] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.801097 14138 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.801244 14138 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.801398 14181 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:23.801472 14181 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:23.801671 14138 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.801877 14181 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.802011 14181 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.802042 14178 tablet_bootstrap.cc:492] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.802070 14181 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.802181 14178 ts_tablet_manager.cc:1367] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:23.802250 14180 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:23.802217 14181 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.802572 14181 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.802637 14178 raft_consensus.cc:348] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.802752 14178 raft_consensus.cc:374] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.802798 14178 raft_consensus.cc:729] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803155 14118 tablet_bootstrap.cc:492] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:23.802382 14180 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:23.803237 14118 ts_tablet_manager.cc:1367] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:23.803640 14118 raft_consensus.cc:348] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.803705 14118 raft_consensus.cc:374] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.803723 14118 raft_consensus.cc:729] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803663 14180 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.803755 14180 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.803787 14180 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.803808 14118 consensus_queue.cc:260] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.803917 14180 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.804008 14118 ts_tablet_manager.cc:1398] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.804191 14180 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:23.805323 14178 consensus_queue.cc:260] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.805665 14178 ts_tablet_manager.cc:1398] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:23.805859 14178 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:23.806788 14178 tablet_bootstrap.cc:654] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:23.807799 14178 tablet_bootstrap.cc:492] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:23.807875 14178 ts_tablet_manager.cc:1367] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:23.808197 14178 raft_consensus.cc:348] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.808310 14178 raft_consensus.cc:374] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:23.808348 14178 raft_consensus.cc:729] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:23.808454 14178 consensus_queue.cc:260] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.808707 14178 ts_tablet_manager.cc:1398] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:23.855181 14179 raft_consensus.cc:482] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.855266 14179 raft_consensus.cc:504] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.855651 14179 leader_election.cc:288] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.855952 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:23.855957 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.856067 8581 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.856067 8975 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.856323 6363 leader_election.cc:302] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.856482 14179 raft_consensus.cc:2781] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.856535 14179 raft_consensus.cc:482] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.856573 14179 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.857582 14179 raft_consensus.cc:504] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.857918 14179 leader_election.cc:288] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.858136 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:23.858233 8581 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.858237 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b43f6d98d9429390c74c722bc1d4d0" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.858351 8975 raft_consensus.cc:3037] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.858974 8581 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.859032 8975 raft_consensus.cc:2454] T a4b43f6d98d9429390c74c722bc1d4d0 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.859205 6363 leader_election.cc:302] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.859397 14179 raft_consensus.cc:2781] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.859475 14179 raft_consensus.cc:686] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:23.859611 14179 consensus_queue.cc:237] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:23.861539 6154 catalog_manager.cc:5484] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:23.864531 14179 raft_consensus.cc:482] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.864609 14179 raft_consensus.cc:504] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.864946 14179 leader_election.cc:288] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:23.865188 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.865283 8975 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.865260 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:23.865418 6559 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:23.865499 6365 leader_election.cc:302] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.865645 14179 raft_consensus.cc:2781] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.865700 14179 raft_consensus.cc:482] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.865739 14179 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.866480 14179 raft_consensus.cc:504] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.866848 14179 leader_election.cc:288] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:23.867085 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.867127 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc4f65253d34507ad704a3d88f0cfbb" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:23.867194 8975 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.867231 6559 raft_consensus.cc:3037] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.867892 8975 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.868028 6559 raft_consensus.cc:2454] T ccc4f65253d34507ad704a3d88f0cfbb P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:23.868202 6365 leader_election.cc:302] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:23.868359 14179 raft_consensus.cc:2781] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.868448 14179 raft_consensus.cc:686] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:23.868570 14179 consensus_queue.cc:237] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:23.870329 6154 catalog_manager.cc:5484] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:23.893488 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.900075 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:23.903124 8975 raft_consensus.cc:1261] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.903199 8843 raft_consensus.cc:1261] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:23.903575 14146 consensus_queue.cc:1035] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.903717 14170 consensus_queue.cc:1035] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:23.905850 14008 mvcc.cc:205] Tried to move back new op lower bound from 7023506079344590848 to 7023506078700249088. Current Snapshot: MvccSnapshot[applied={T|T < 7023506079344590848}]
I20240503 07:59:23.906531 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.907151 14194 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:23.907435 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.907737 6155 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.907872 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.908061 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357]). Expected version 1 got 0
I20240503 07:59:23.908200 14195 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:23.910342 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:23.912112 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 1)
I20240503 07:59:23.912353 14193 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
W20240503 07:59:23.937774 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:23.941078 14124 raft_consensus.cc:482] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:23.941161 14124 raft_consensus.cc:504] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.941468 14124 leader_election.cc:288] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.941766 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:23.941766 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:23.941896 8975 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.941896 8581 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:23.942194 6232 leader_election.cc:302] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:23.942395 14124 raft_consensus.cc:2781] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:23.942462 14124 raft_consensus.cc:482] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:23.942507 14124 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.943408 14124 raft_consensus.cc:504] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.943783 14124 leader_election.cc:288] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:23.944092 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:23.944095 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56117fc1ae74473b3836584a22b6b16" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:23.944191 8975 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.944195 8581 raft_consensus.cc:3037] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:23.945104 8975 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.945120 8581 raft_consensus.cc:2454] T c56117fc1ae74473b3836584a22b6b16 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:23.945403 6234 leader_election.cc:302] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:23.945590 14124 raft_consensus.cc:2781] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:23.945700 14124 raft_consensus.cc:686] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:23.945850 14124 consensus_queue.cc:237] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:23.947878 6155 catalog_manager.cc:5484] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:23.976306 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:23.997613 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.015707 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.020413 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.020648 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.020718 14194 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.020833 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.021157 14193 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:24.021179 14195 tablet.cc:1620] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:24.025437 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 2)
I20240503 07:59:24.128479 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:24.133468 14193 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.133967 14194 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.135344 14195 tablet.cc:1722] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.140028 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357] alter complete (version 3)
W20240503 07:59:24.144037 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:24.152984 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:24.174988 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.243746 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.244016 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.246352 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.246896 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:24.246925 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:24.247172 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ca18d3d275477f832ccd94b1958924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:24.247349 14200 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:24.247383 14199 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:24.247464 14201 tablet_replica.cc:330] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:24.247483 14200 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.247537 14199 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.247609 14201 raft_consensus.cc:2229] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:24.247646 14199 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.247642 14200 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.247840 14201 raft_consensus.cc:2258] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.248234 14199 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.248577 14200 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.249198 14201 ts_tablet_manager.cc:1875] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.250945 14199 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.251055 14199 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.251325 14199 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:24.251897 14201 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.251976 14201 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.252249 14201 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:24.253146 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
I20240503 07:59:24.255262 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
W20240503 07:59:24.256101 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:24.256932 14200 ts_tablet_manager.cc:1888] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.257050 14200 log.cc:1196] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/01ca18d3d275477f832ccd94b1958924
I20240503 07:59:24.257628 14200 ts_tablet_manager.cc:1909] T 01ca18d3d275477f832ccd94b1958924 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:24.258281 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 01ca18d3d275477f832ccd94b1958924 (table rebalancer_dynamic_table_new_name [id=a4fae4cc5cb8496fb1900c2c36767357]) successfully deleted
I20240503 07:59:24.274338 14179 consensus_queue.cc:1035] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.278115 14204 consensus_queue.cc:1035] T a4b43f6d98d9429390c74c722bc1d4d0 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:24.284508 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:24.339026 14063 consensus_queue.cc:1035] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.341678 14063 consensus_queue.cc:1035] T c56117fc1ae74473b3836584a22b6b16 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.347896 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:24.352699 6539 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.352983 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.352964 8824 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.353169 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:24.355262 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:24.355491 14212 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.355468 6279 tablet_service.cc:1467] Processing CreateTablet for tablet a42143d78f0b45e485cc70c357652b2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]), partition=
I20240503 07:59:24.355676 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a42143d78f0b45e485cc70c357652b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.356072 14211 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:24.356462 14212 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.357008 14211 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.358395 14212 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.358544 14212 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.358927 14215 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:24.358959 14212 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.359097 14212 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.359155 14212 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.359289 14212 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.359472 14211 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:24.359567 14212 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.359584 14211 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.359988 14211 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.360126 14211 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.360191 14211 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.360414 14215 tablet_bootstrap.cc:654] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.360357 14211 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.361385 14211 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.361927 14215 tablet_bootstrap.cc:492] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:24.362124 14215 ts_tablet_manager.cc:1367] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.362504 14215 raft_consensus.cc:348] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.362655 14215 raft_consensus.cc:374] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.362727 14215 raft_consensus.cc:729] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.362887 14215 consensus_queue.cc:260] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.363322 14215 ts_tablet_manager.cc:1398] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:24.386463 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.398824 14204 consensus_queue.cc:1035] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.401899 14109 consensus_queue.cc:1035] T ccc4f65253d34507ad704a3d88f0cfbb P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:24.477234 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.478379 14168 raft_consensus.cc:482] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.478463 14168 raft_consensus.cc:504] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.478781 14168 leader_election.cc:288] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:24.479139 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:24.479183 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:24.479292 6560 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.479298 6296 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.479537 8776 leader_election.cc:302] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:24.479713 14168 raft_consensus.cc:2781] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.479782 14168 raft_consensus.cc:482] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.479822 14168 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.480655 14168 raft_consensus.cc:504] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.480994 14168 leader_election.cc:288] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:24.481323 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:24.481367 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a42143d78f0b45e485cc70c357652b2c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:24.481446 6560 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.481458 6296 raft_consensus.cc:3037] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.481788 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:24.482415 6560 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.482414 6296 raft_consensus.cc:2454] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.482754 8776 leader_election.cc:302] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:24.482921 14168 raft_consensus.cc:2781] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.483032 14168 raft_consensus.cc:686] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:24.483179 14168 consensus_queue.cc:237] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:24.487632 6155 catalog_manager.cc:5484] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.488113 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.488256 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.488371 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.488529 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.490689 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 302fd0d601384c7897e71cf022bc1f1b (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490810 14222 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:24.490808 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490808 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.490926 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fd0d601384c7897e71cf022bc1f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491046 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491047 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.491091 14221 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.493046 14212 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.493067 14222 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.493319 14221 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.493328 8822 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.493556 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.493790 14224 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:24.493968 14212 tablet_bootstrap.cc:654] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.494659 14224 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.494776 14221 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.494859 14221 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.495280 14221 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.495369 14221 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.495397 14221 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.495406 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d74da8f3fd1b46028675a04564734ced (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.495486 14221 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.495618 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74da8f3fd1b46028675a04564734ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.495638 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.495739 14221 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.495987 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.497591 14221 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.498025 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf198f24c2f4e5ba43f1fe0205469f2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2da91173843f4852b7eabd3e3d0336d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:24.498262 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf198f24c2f4e5ba43f1fe0205469f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:24.498415 14221 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.498629 14212 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.498682 14212 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:24.499032 14212 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.499135 14212 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.499174 14212 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.499287 14212 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.499434 14221 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.499519 14221 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:24.499540 14212 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.499677 14212 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:24.499907 14221 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.500011 14221 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.500052 14221 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.500167 14221 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.500324 14211 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:24.500380 14212 tablet_bootstrap.cc:654] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.500414 14221 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.500551 14221 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:24.501209 14211 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.501304 14223 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:24.501353 14221 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.502457 14223 tablet_bootstrap.cc:654] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:24.502758 14212 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:24.502835 14212 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:24.503216 14212 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.503297 14212 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.503337 14212 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.503435 14212 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.503666 14222 tablet_bootstrap.cc:492] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:24.503666 14212 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.503737 14222 ts_tablet_manager.cc:1367] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:24.504016 14221 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:24.504124 14222 raft_consensus.cc:348] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.504228 14222 raft_consensus.cc:374] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.504238 14221 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:24.504264 14222 raft_consensus.cc:729] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.504370 14222 consensus_queue.cc:260] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.504611 14222 ts_tablet_manager.cc:1398] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.504627 14221 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.504722 14221 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.504752 14221 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.505400 14221 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.505733 14221 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.506701 14211 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:24.506780 14211 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240503 07:59:24.507184 14211 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.507285 14211 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.507314 14211 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.507412 14211 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.507651 14211 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:24.507768 14224 tablet_bootstrap.cc:492] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:24.507848 14224 ts_tablet_manager.cc:1367] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:24.507978 14223 tablet_bootstrap.cc:492] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:24.508049 14223 ts_tablet_manager.cc:1367] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:24.508266 14224 raft_consensus.cc:348] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.508591 14224 raft_consensus.cc:374] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.508625 14224 raft_consensus.cc:729] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.508863 14224 consensus_queue.cc:260] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.509140 14224 ts_tablet_manager.cc:1398] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:24.508452 14223 raft_consensus.cc:348] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.511906 14223 raft_consensus.cc:374] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:24.511957 14223 raft_consensus.cc:729] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:24.512066 14223 consensus_queue.cc:260] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.512292 14223 ts_tablet_manager.cc:1398] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:24.536136 14168 raft_consensus.cc:482] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.536228 14168 raft_consensus.cc:504] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.536584 14168 leader_election.cc:288] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:24.536862 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:24.536917 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:24.536995 6690 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.537025 6429 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:24.537285 8779 leader_election.cc:302] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:24.537474 14174 raft_consensus.cc:2781] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.537534 14174 raft_consensus.cc:482] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.537572 14174 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.538463 14174 raft_consensus.cc:504] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.538789 14174 leader_election.cc:288] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:24.539076 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:24.539120 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74da8f3fd1b46028675a04564734ced" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:24.539178 6690 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.539201 6429 raft_consensus.cc:3037] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.539958 6429 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.540015 6690 raft_consensus.cc:2454] T d74da8f3fd1b46028675a04564734ced P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:24.540277 8778 leader_election.cc:302] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.540488 14174 raft_consensus.cc:2781] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.540563 14174 raft_consensus.cc:686] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:24.540705 14174 consensus_queue.cc:237] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.543228 6155 catalog_manager.cc:5484] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:24.546839 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.575613 14197 raft_consensus.cc:482] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.575724 14197 raft_consensus.cc:504] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.576092 14197 leader_election.cc:288] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:24.576323 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:24.576450 8581 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:24.576463 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:24.576567 8842 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:24.576737 6625 leader_election.cc:302] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:24.576931 14197 raft_consensus.cc:2781] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.576984 14197 raft_consensus.cc:482] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.577041 14106 raft_consensus.cc:482] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:24.577103 14197 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.577121 14106 raft_consensus.cc:504] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.577389 14106 leader_election.cc:288] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:24.577697 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:24.577757 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:24.577826 6690 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:24.577883 6560 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:24.578117 8912 leader_election.cc:302] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.578146 14197 raft_consensus.cc:504] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.578451 14197 leader_election.cc:288] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:24.578485 14106 raft_consensus.cc:2781] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:24.578536 14106 raft_consensus.cc:482] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:24.578570 14106 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.578687 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:24.578773 8581 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.579464 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fd0d601384c7897e71cf022bc1f1b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:24.579576 8842 raft_consensus.cc:3037] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.579555 14106 raft_consensus.cc:504] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.579607 8581 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:24.579869 14106 leader_election.cc:288] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:24.579849 6625 leader_election.cc:302] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:24.580004 14197 raft_consensus.cc:2781] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.580199 14197 raft_consensus.cc:686] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:24.580305 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:24.580336 14197 consensus_queue.cc:237] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:24.580408 6690 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.580394 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cf198f24c2f4e5ba43f1fe0205469f2" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:24.580489 6560 raft_consensus.cc:3037] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:24.580524 8842 raft_consensus.cc:2454] T 302fd0d601384c7897e71cf022bc1f1b P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:24.581488 6690 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:24.581502 6560 raft_consensus.cc:2454] T 9cf198f24c2f4e5ba43f1fe0205469f2 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:24.581743 8912 leader_election.cc:302] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:24.582321 14106 raft_consensus.cc:2781] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:24.582515 14106 raft_consensus.cc:686] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:24.582648 14106 consensus_queue.cc:237] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:24.583223 6155 catalog_manager.cc:5484] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:24.584947 6156 catalog_manager.cc:5484] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:24.605856 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:24.609153 6560 raft_consensus.cc:1261] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.609153 6296 raft_consensus.cc:1261] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:24.609511 14174 consensus_queue.cc:1035] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.609660 14168 consensus_queue.cc:1035] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:24.612165 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.612334 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]). Expected version 1 got 0
I20240503 07:59:24.612458 14194 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.613233 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4]). Expected version 1 got 0
I20240503 07:59:24.616619 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 1)
I20240503 07:59:24.616900 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.617666 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:24.617437 14240 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.620061 14022 mvcc.cc:205] Tried to move back new op lower bound from 7023506082235392000 to 7023506081724542976. Current Snapshot: MvccSnapshot[applied={T|T < 7023506082235392000}]
I20240503 07:59:24.617954 14241 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
W20240503 07:59:24.701854 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.718534 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:24.723470 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.723784 14194 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:24.724006 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.724435 14241 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:24.724607 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:24.724923 14240 tablet.cc:1620] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:24.729859 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 2)
W20240503 07:59:24.749282 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:24.788164 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.808043 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.832820 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240503 07:59:24.834463 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.837617 14194 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.837913 14241 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:24.838910 14240 tablet.cc:1722] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:24.843109 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:24.844004 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4] alter complete (version 3)
W20240503 07:59:24.875711 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.897498 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:24.939028 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240503 07:59:24.946171 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.946300 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:24.948482 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.948964 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:24.948964 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:24.949074 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet a42143d78f0b45e485cc70c357652b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:24 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:24.949378 14249 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:24.949419 14247 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:24.949497 14249 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.949573 14247 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:24.949613 14249 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.949750 14247 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.949977 14248 tablet_replica.cc:330] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:24.950119 14248 raft_consensus.cc:2229] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:24.950241 14248 raft_consensus.cc:2258] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:24.950263 14249 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.950264 14247 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.950850 14248 ts_tablet_manager.cc:1875] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:24.952242 14249 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.952323 14249 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.952477 14247 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.952553 14247 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.952664 14249 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:24.952862 14247 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:24.953094 14248 ts_tablet_manager.cc:1888] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:24.953171 14248 log.cc:1196] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a42143d78f0b45e485cc70c357652b2c
I20240503 07:59:24.953356 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
I20240503 07:59:24.953493 14248 ts_tablet_manager.cc:1909] T a42143d78f0b45e485cc70c357652b2c P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:24.953665 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
I20240503 07:59:24.954052 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet a42143d78f0b45e485cc70c357652b2c (table rebalancer_dynamic_table_new_name [id=e8f5f9d82bb644a8b6f446b560fafbd4]) successfully deleted
W20240503 07:59:24.987248 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 07:59:24.993680 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 07:59:25.034438 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 07:59:25.044414 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.046444 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240503 07:59:25.050552 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:25.051028 14197 consensus_queue.cc:1035] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.054541 14236 consensus_queue.cc:1035] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.056849 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.056833 14197 consensus_queue.cc:1035] T 302fd0d601384c7897e71cf022bc1f1b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.057080 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.056741 6539 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.058595 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.059300 14256 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.059496 8560 tablet_service.cc:1467] Processing CreateTablet for tablet bcd2544c453d42b3acc392f4929c0bdf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]), partition=
I20240503 07:59:25.059705 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd2544c453d42b3acc392f4929c0bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.060494 14256 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.062611 14259 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.063511 14259 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.065279 14257 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.066358 14257 tablet_bootstrap.cc:654] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.066689 14236 consensus_queue.cc:1035] T 9cf198f24c2f4e5ba43f1fe0205469f2 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.067581 14257 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:25.067706 14257 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:25.067986 14259 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.068125 14259 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:25.068176 14257 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.068274 14257 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.068307 14257 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.068434 14257 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.068694 14257 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.068881 14259 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.069031 14259 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.069108 14259 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.069263 14259 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.069582 14259 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.073278 14256 tablet_bootstrap.cc:492] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.073401 14256 ts_tablet_manager.cc:1367] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:25.073989 14256 raft_consensus.cc:348] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.074106 14256 raft_consensus.cc:374] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.074146 14256 raft_consensus.cc:729] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.074255 14256 consensus_queue.cc:260] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.074507 14256 ts_tablet_manager.cc:1398] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.085692 14168 consensus_queue.cc:1035] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.088450 14174 consensus_queue.cc:1035] T d74da8f3fd1b46028675a04564734ced P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.101797 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:25.106851 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 07:59:25.107749 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.107720 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.107998 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.108079 6408 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.108083 8957 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.108279 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.108330 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.111019 14268 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:25.110997 8822 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.111245 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.111835 14268 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.112921 14268 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:25.112980 14268 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.113399 14256 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.113399 14268 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.113493 14268 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.113524 14268 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.113636 14268 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.113986 14268 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:25.114035 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 07:59:25.114421 14256 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.114732 6279 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.114812 8560 tablet_service.cc:1467] Processing CreateTablet for tablet c6b931be92c34169a9f1f0a6146fbb0a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.114944 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.115088 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6b931be92c34169a9f1f0a6146fbb0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.116456 14256 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.116551 14256 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.116849 14272 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.117031 14256 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.117136 14256 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.117167 14256 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.117286 14259 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.117277 14256 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.117857 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.118067 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.118237 6277 tablet_service.cc:1467] Processing CreateTablet for tablet ab614f5e420142edaed13d6997e77fab (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.118321 14259 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.118419 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab614f5e420142edaed13d6997e77fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.118674 14256 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.119079 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2a0cc7a2004f968226731f04b71835 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=b2ea93bb709c4406851b78d99cf9d162]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.119288 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2a0cc7a2004f968226731f04b71835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.119400 14259 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.119470 14259 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:25.119902 14259 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.120013 14259 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.120047 14259 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.120167 14259 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.120458 14259 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:25.120827 14269 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.121495 14272 tablet_bootstrap.cc:654] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.123064 14259 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.123142 14269 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.124222 14259 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:25.124986 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.125480 14259 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:25.125566 14259 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:25.125936 14269 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.125921 14259 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.125991 14269 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:25.125988 14259 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.126013 14259 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.126112 14259 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.126318 14269 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.126368 14259 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.126392 14269 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.126412 14269 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.126518 14269 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.126856 14269 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.127167 14272 tablet_bootstrap.cc:492] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.127242 14272 ts_tablet_manager.cc:1367] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:25.127643 14272 raft_consensus.cc:348] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.127734 14272 raft_consensus.cc:374] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.127764 14272 raft_consensus.cc:729] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.127867 14269 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.127866 14272 consensus_queue.cc:260] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.128120 14272 ts_tablet_manager.cc:1398] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.128262 14272 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.128805 14269 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.129067 14272 tablet_bootstrap.cc:654] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.132123 14269 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.132238 14269 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:25.132990 14269 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.133133 14269 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.133208 14269 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.133447 14269 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.134438 14269 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.135030 14272 tablet_bootstrap.cc:492] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.135118 14272 ts_tablet_manager.cc:1367] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:25.135473 14272 raft_consensus.cc:348] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.135560 14272 raft_consensus.cc:374] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.135591 14272 raft_consensus.cc:729] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.135710 14272 consensus_queue.cc:260] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.135985 14272 ts_tablet_manager.cc:1398] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.136183 14272 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:25.137082 14272 tablet_bootstrap.cc:654] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:25.138739 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.139338 14272 tablet_bootstrap.cc:492] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:25.139417 14272 ts_tablet_manager.cc:1367] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.139827 14272 raft_consensus.cc:348] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.139910 14272 raft_consensus.cc:374] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.139942 14272 raft_consensus.cc:729] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.140060 14272 consensus_queue.cc:260] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.140283 14272 ts_tablet_manager.cc:1398] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:25.148039 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:25.162971 14243 raft_consensus.cc:482] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.163064 14243 raft_consensus.cc:504] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.163400 14243 leader_election.cc:288] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.163664 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.163704 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.163790 8843 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:25.163805 8581 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:25.164098 6496 leader_election.cc:302] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:25.164461 14243 raft_consensus.cc:2781] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.164528 14243 raft_consensus.cc:482] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.164570 14243 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.165637 14243 raft_consensus.cc:504] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.165971 14243 leader_election.cc:288] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.166486 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.166589 8843 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.166594 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd2544c453d42b3acc392f4929c0bdf" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.166678 8581 raft_consensus.cc:3037] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.167479 8581 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:25.167479 8843 raft_consensus.cc:2454] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:25.167821 6494 leader_election.cc:302] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.168282 14243 raft_consensus.cc:2781] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.168363 14243 raft_consensus.cc:686] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:25.168485 14243 consensus_queue.cc:237] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.170495 6156 catalog_manager.cc:5484] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.178460 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 07:59:25.179116 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:25.195617 14245 raft_consensus.cc:482] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.195703 14245 raft_consensus.cc:504] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.196017 14245 leader_election.cc:288] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.196288 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.196328 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:25.196394 8581 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.196471 8976 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.196628 6232 leader_election.cc:302] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.196928 14245 raft_consensus.cc:2781] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.196979 14245 raft_consensus.cc:482] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.197005 14245 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.197770 14245 raft_consensus.cc:504] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.198050 14245 leader_election.cc:288] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.198340 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:25.198355 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6b931be92c34169a9f1f0a6146fbb0a" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.198439 8581 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.198441 8976 raft_consensus.cc:3037] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.199187 8581 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.199218 8976 raft_consensus.cc:2454] T c6b931be92c34169a9f1f0a6146fbb0a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.199424 6232 leader_election.cc:302] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:25.199560 14245 raft_consensus.cc:2781] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.199646 14245 raft_consensus.cc:686] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:25.199765 14245 consensus_queue.cc:237] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.201882 6154 catalog_manager.cc:5484] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.239369 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.268378 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.279824 14245 raft_consensus.cc:482] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.279918 14245 raft_consensus.cc:504] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.280264 14245 leader_election.cc:288] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:25.280519 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.280505 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:25.280625 8843 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.280627 6430 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:25.280886 6235 leader_election.cc:302] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.281077 14245 raft_consensus.cc:2781] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.281137 14245 raft_consensus.cc:482] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.281163 14245 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.281898 14245 raft_consensus.cc:504] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.282166 14245 leader_election.cc:288] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:25.282438 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.282431 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab614f5e420142edaed13d6997e77fab" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:25.282555 8843 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.282538 6430 raft_consensus.cc:3037] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.283413 6430 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.283413 8843 raft_consensus.cc:2454] T ab614f5e420142edaed13d6997e77fab P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:25.283674 6234 leader_election.cc:302] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:25.283833 14245 raft_consensus.cc:2781] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.283951 14245 raft_consensus.cc:686] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:25.284085 14245 consensus_queue.cc:237] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:25.286360 6154 catalog_manager.cc:5484] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.293584 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:25.296885 8843 raft_consensus.cc:1261] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.296911 8581 raft_consensus.cc:1261] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:25.297214 13931 consensus_queue.cc:1035] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.297405 14243 consensus_queue.cc:1035] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.300230 6154 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]). Expected version 1 got 0
I20240503 07:59:25.300283 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.300594 14240 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.301252 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.301618 14286 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:25.302466 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43]). Expected version 1 got 0
I20240503 07:59:25.304359 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:25.304622 14194 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:25.304998 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43] alter complete (version 1)
W20240503 07:59:25.343437 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 07:59:25.409979 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:25.415477 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.415791 14240 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:25.416134 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.416235 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:25.416461 14194 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:25.416687 14286 tablet.cc:1620] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:25.421741 14179 raft_consensus.cc:482] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.421888 14179 raft_consensus.cc:504] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.422243 14179 leader_election.cc:288] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.422667 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:25.422828 6298 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:25.422955 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:25.423067 8581 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:25.423274 6363 leader_election.cc:302] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.423595 14179 raft_consensus.cc:2781] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.423709 14179 raft_consensus.cc:482] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.423807 14179 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.425302 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=008a18121d074d329f504665d5dfcc43] alter complete (version 2)
I20240503 07:59:25.425459 14179 raft_consensus.cc:504] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.426477 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:25.426594 6298 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.426716 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2a0cc7a2004f968226731f04b71835" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:25.426813 8581 raft_consensus.cc:3037] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.427562 6298 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:25.427707 8581 raft_consensus.cc:2454] T 8c2a0cc7a2004f968226731f04b71835 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:25.428066 6363 leader_election.cc:302] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:25.428303 14204 raft_consensus.cc:2781] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.428471 14204 raft_consensus.cc:686] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:25.428615 14204 consensus_queue.cc:237] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:25.428926 14179 leader_election.cc:288] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:25.433759 6156 catalog_manager.cc:5484] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:25.466337 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:25.483806 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:25.528755 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:25.533466 14240 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.534278 14194 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.536854 14286 tablet.cc:1722] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:25.538676 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43] alter complete (version 3)
W20240503 07:59:25.552088 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:25.593273 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 07:59:25.615561 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:25.627889 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): CreateTablet RPC failed for tablet ef76600ded194f318cf97f157845bea8: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:25.640537 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.640667 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:25.642556 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.643023 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:25.643035 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:25.643374 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd2544c453d42b3acc392f4929c0bdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:25 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:25.643404 14295 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:25.643417 14296 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:25.643530 14296 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:25.643530 14295 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.643615 14297 tablet_replica.cc:330] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:25.643640 14295 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.643786 14297 raft_consensus.cc:2229] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:25.643847 14296 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.643946 14297 raft_consensus.cc:2258] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:25.644424 14295 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.644579 14296 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.644593 14297 ts_tablet_manager.cc:1875] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:25.646550 14297 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646657 14297 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646770 14296 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646770 14295 ts_tablet_manager.cc:1888] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:25.646842 14295 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646888 14296 log.cc:1196] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/bcd2544c453d42b3acc392f4929c0bdf
I20240503 07:59:25.646924 14297 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:25.647099 14295 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:25.647166 14296 ts_tablet_manager.cc:1909] T bcd2544c453d42b3acc392f4929c0bdf P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:25.647541 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.647701 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.647722 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet bcd2544c453d42b3acc392f4929c0bdf (table rebalancer_dynamic_table_new_name [id=008a18121d074d329f504665d5dfcc43]) successfully deleted
I20240503 07:59:25.723004 14244 consensus_queue.cc:1035] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.726688 14245 consensus_queue.cc:1035] T c6b931be92c34169a9f1f0a6146fbb0a P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.744082 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:25.748764 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.748916 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.749027 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.749168 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.751008 14302 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:25.751251 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3f857930d542fa8a07311090c88e22 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]), partition=
I20240503 07:59:25.751302 14303 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:25.751489 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b3f857930d542fa8a07311090c88e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.752086 14302 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.752113 14303 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.752947 14302 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:25.753006 14302 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:25.753324 14302 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.753397 14302 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.753420 14302 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.753525 14302 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.753571 14306 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.753787 14302 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:25.754317 14303 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:25.754473 14303 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.754561 14306 tablet_bootstrap.cc:654] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.754849 14303 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.754935 14303 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.754962 14303 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.755087 14303 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.755326 14303 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.756201 14306 tablet_bootstrap.cc:492] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:25.756285 14306 ts_tablet_manager.cc:1367] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.756701 14306 raft_consensus.cc:348] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.756886 14306 raft_consensus.cc:374] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:25.756939 14306 raft_consensus.cc:729] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:25.757061 14306 consensus_queue.cc:260] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.757401 14306 ts_tablet_manager.cc:1398] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:25.797529 14244 consensus_queue.cc:1035] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.800944 14244 consensus_queue.cc:1035] T ab614f5e420142edaed13d6997e77fab P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.885898 14289 raft_consensus.cc:482] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:25.886020 14289 raft_consensus.cc:504] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.886505 14289 leader_election.cc:288] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:25.886763 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:25.886888 8843 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:25.887008 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:25.887121 6627 leader_election.cc:302] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:25.887178 6559 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:25.887259 14289 raft_consensus.cc:2781] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:25.887308 14289 raft_consensus.cc:482] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:25.887341 14289 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.888150 14289 raft_consensus.cc:504] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.888451 14289 leader_election.cc:288] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:25.888695 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:25.888854 8843 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.888875 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3f857930d542fa8a07311090c88e22" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:25.888998 6559 raft_consensus.cc:3037] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:25.889868 8843 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:25.889948 6559 raft_consensus.cc:2454] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:25.890156 6627 leader_election.cc:302] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:25.890406 14289 raft_consensus.cc:2781] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:25.890480 14289 raft_consensus.cc:686] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:25.890595 14289 consensus_queue.cc:237] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:25.892750 6156 catalog_manager.cc:5484] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:25.908703 14204 consensus_queue.cc:1035] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.911408 14204 consensus_queue.cc:1035] T 8c2a0cc7a2004f968226731f04b71835 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:25.983474 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:25.990603 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.990796 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.990942 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.991070 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.991276 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5b794d73f54463a25c52b49f8dee71 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.991544 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5b794d73f54463a25c52b49f8dee71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.993647 14315 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:25.993686 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.993880 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.994891 14315 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.995529 14316 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:25.995744 14306 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:25.996371 14316 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.996556 14306 tablet_bootstrap.cc:654] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:25.996995 14315 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:25.997067 14315 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:25.997723 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997824 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 677a26a61d7547d59aa08ecc5a1d64d4 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997881 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.997992 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.997982 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:25.998071 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677a26a61d7547d59aa08ecc5a1d64d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.998220 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.998268 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:25.997519 14315 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.000460 14303 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.000469 14302 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.000805 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 0df75a72a28349458c7cf145ad9b4715 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=d9f27007be7e4db186393a865adbd54a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.001016 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df75a72a28349458c7cf145ad9b4715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.001339 14302 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.001386 14303 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.001392 14306 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.001458 14306 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:26.001873 14306 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.001991 14306 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.002024 14306 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.002130 14306 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.002214 14315 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.002421 14306 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.002429 14315 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.002557 14315 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.003275 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:26.003414 14315 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:26.004423 14302 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.004514 14302 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:26.005034 14302 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.005133 14302 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.005154 14302 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.005249 14302 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.005504 14302 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.005661 14302 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.006455 14302 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.007603 14316 tablet_bootstrap.cc:492] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.007680 14316 ts_tablet_manager.cc:1367] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:26.009286 14316 raft_consensus.cc:348] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.009385 14316 raft_consensus.cc:374] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.009416 14316 raft_consensus.cc:729] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.009528 14316 consensus_queue.cc:260] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.009792 14316 ts_tablet_manager.cc:1398] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.009932 14316 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:26.011126 14316 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.013111 14302 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.013190 14302 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:26.013566 8843 raft_consensus.cc:1261] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.013655 6559 raft_consensus.cc:1261] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.013689 14302 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.013772 14302 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.013797 14302 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.013906 14302 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.013911 14197 consensus_queue.cc:1035] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.014058 14251 consensus_queue.cc:1035] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.014146 14302 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.017323 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.017594 14194 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.019572 6156 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]). Expected version 1 got 0
I20240503 07:59:26.020120 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.020416 14328 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.021679 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd]). Expected version 1 got 0
I20240503 07:59:26.023641 14303 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.023676 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd] alter complete (version 1)
I20240503 07:59:26.023726 14303 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 07:59:26.023861 14319 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:26.024138 14303 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.024250 14303 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.024289 14303 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.024399 14303 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.024681 14303 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.029017 14319 tablet_bootstrap.cc:654] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.030264 14319 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:26.030361 14319 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:26.030475 14316 tablet_bootstrap.cc:492] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.030558 14316 ts_tablet_manager.cc:1367] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:26.030807 14319 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.030928 14319 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.030972 14319 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.030941 14316 raft_consensus.cc:348] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031041 14316 raft_consensus.cc:374] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.031075 14316 raft_consensus.cc:729] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.031183 14316 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031188 14319 consensus_queue.cc:260] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.031430 14316 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.031514 14319 ts_tablet_manager.cc:1398] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.031664 14319 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:26.032470 14319 tablet_bootstrap.cc:654] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.033273 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.033545 14240 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.053018 14319 tablet_bootstrap.cc:492] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:26.053117 14319 ts_tablet_manager.cc:1367] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 07:59:26.053476 14319 raft_consensus.cc:348] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.053567 14319 raft_consensus.cc:374] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.053596 14319 raft_consensus.cc:729] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.053714 14319 consensus_queue.cc:260] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.053951 14319 ts_tablet_manager.cc:1398] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.070333 14291 raft_consensus.cc:482] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.070433 14291 raft_consensus.cc:504] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.070878 14291 leader_election.cc:288] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.071169 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:26.071224 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.071282 8975 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.071352 8581 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.071682 8778 leader_election.cc:302] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:26.071934 14291 raft_consensus.cc:2781] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.072003 14291 raft_consensus.cc:482] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.072036 14291 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.073009 14291 raft_consensus.cc:504] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.073341 14291 leader_election.cc:288] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.073690 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:26.073691 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677a26a61d7547d59aa08ecc5a1d64d4" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.073799 8581 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.073799 8975 raft_consensus.cc:3037] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.074684 8581 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.074684 8975 raft_consensus.cc:2454] T 677a26a61d7547d59aa08ecc5a1d64d4 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.074942 8776 leader_election.cc:302] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.075091 14291 raft_consensus.cc:2781] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.075232 14291 raft_consensus.cc:686] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.075361 14291 consensus_queue.cc:237] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:26.077214 6155 catalog_manager.cc:5484] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.125918 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:26.130899 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.131218 14328 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.131282 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.131556 14194 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.133836 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.134127 14240 tablet.cc:1620] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.136188 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb43c22ada004e5490840b44458921dd] alter complete (version 2)
I20240503 07:59:26.147596 14243 raft_consensus.cc:482] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.147689 14243 raft_consensus.cc:504] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.148269 14243 leader_election.cc:288] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.148471 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:26.148619 6430 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.148975 6497 leader_election.cc:302] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.148975 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.149092 8581 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.149247 13931 raft_consensus.cc:2781] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.149302 13931 raft_consensus.cc:482] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.149355 13931 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.150369 13931 raft_consensus.cc:504] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.150730 13931 leader_election.cc:288] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:26.151057 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.151168 8581 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.151175 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5b794d73f54463a25c52b49f8dee71" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:26.151296 6430 raft_consensus.cc:3037] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.152088 8581 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.152204 6430 raft_consensus.cc:2454] T 6a5b794d73f54463a25c52b49f8dee71 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.152324 6494 leader_election.cc:302] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.152580 13931 raft_consensus.cc:2781] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.152668 13931 raft_consensus.cc:686] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:26.152865 13931 consensus_queue.cc:237] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:26.154770 6154 catalog_manager.cc:5484] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:26.163216 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.163662 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:26.173126 14291 raft_consensus.cc:482] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.173221 14291 raft_consensus.cc:504] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.173542 14291 leader_election.cc:288] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:26.173801 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:26.173858 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:26.173923 6691 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.173970 8975 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.174175 8778 leader_election.cc:302] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:26.174332 14291 raft_consensus.cc:2781] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.174391 14291 raft_consensus.cc:482] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.174435 14291 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.175112 14291 raft_consensus.cc:504] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.175381 14291 leader_election.cc:288] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:26.175652 6691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:26.175684 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df75a72a28349458c7cf145ad9b4715" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:26.175766 6691 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.175769 8975 raft_consensus.cc:3037] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.176432 6691 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.176683 8778 leader_election.cc:302] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:26.176748 8975 raft_consensus.cc:2454] T 0df75a72a28349458c7cf145ad9b4715 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.176966 14291 raft_consensus.cc:2781] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.177055 14291 raft_consensus.cc:686] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.177258 14291 consensus_queue.cc:237] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:26.179360 6154 catalog_manager.cc:5484] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.239907 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:26.244695 14328 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.245110 14240 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.245254 14194 tablet.cc:1722] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.250672 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd] alter complete (version 3)
W20240503 07:59:26.289852 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.354197 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.354370 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:26.356508 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.357048 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:26.357122 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:26.357144 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3f857930d542fa8a07311090c88e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:26 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:26.357352 14336 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:26.357439 14337 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:26.357503 14336 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.357517 14338 tablet_replica.cc:330] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:26.357574 14337 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:26.357625 14336 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.357636 14338 raft_consensus.cc:2229] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:26.357736 14338 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.357764 14337 raft_consensus.cc:2258] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:26.358294 14337 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.358295 14338 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.358435 14336 ts_tablet_manager.cc:1875] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:26.360155 14336 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360220 14336 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360389 14338 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360389 14337 ts_tablet_manager.cc:1888] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:26.360430 14336 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:26.360464 14338 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360464 14337 log.cc:1196] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/3b3f857930d542fa8a07311090c88e22
I20240503 07:59:26.360756 14337 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:26.360821 14338 ts_tablet_manager.cc:1909] T 3b3f857930d542fa8a07311090c88e22 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:26.361248 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.361407 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.361567 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 3b3f857930d542fa8a07311090c88e22 (table rebalancer_dynamic_table_new_name [id=bb43c22ada004e5490840b44458921dd]) successfully deleted
I20240503 07:59:26.458215 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 07:59:26.463099 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.463950 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.464254 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.464386 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.464632 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.465162 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 374f4273dd194b3491e8542de68caba3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]), partition=
I20240503 07:59:26.465399 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374f4273dd194b3491e8542de68caba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.466892 14303 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.467109 14306 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:26.467768 14303 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.467842 14306 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.468571 14302 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.469465 14302 tablet_bootstrap.cc:654] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.469528 14306 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.469640 14306 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.470198 14303 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.470124 14306 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470249 14306 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.470263 14303 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:26.470286 14306 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.470422 14306 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470635 14303 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.470721 14303 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.470741 14303 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.470754 14306 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.470839 14303 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.471078 14303 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.472214 14302 tablet_bootstrap.cc:492] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.472297 14302 ts_tablet_manager.cc:1367] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:26.472730 14302 raft_consensus.cc:348] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.473389 14302 raft_consensus.cc:374] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.473431 14302 raft_consensus.cc:729] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.473533 14302 consensus_queue.cc:260] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.473814 14302 ts_tablet_manager.cc:1398] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:26.518272 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.550725 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:26.567178 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:26.568089 14242 consensus_queue.cc:1035] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.568253 14291 consensus_queue.cc:1035] T 677a26a61d7547d59aa08ecc5a1d64d4 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:26.597110 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.628145 14242 raft_consensus.cc:482] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.628262 14242 raft_consensus.cc:504] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.628643 14242 leader_election.cc:288] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:26.628903 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:26.629067 6559 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.629117 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:26.629240 6690 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:26.629397 8776 leader_election.cc:302] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.629592 14242 raft_consensus.cc:2781] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.629637 14242 raft_consensus.cc:482] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.629662 14242 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.630439 14242 raft_consensus.cc:504] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.630702 14242 leader_election.cc:288] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:26.630970 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:26.631062 6559 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.631009 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374f4273dd194b3491e8542de68caba3" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:26.631146 6690 raft_consensus.cc:3037] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.631803 6559 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.631938 6690 raft_consensus.cc:2454] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:26.632055 8776 leader_election.cc:302] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.632200 14242 raft_consensus.cc:2781] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.632308 14242 raft_consensus.cc:686] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:26.632427 14242 consensus_queue.cc:237] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:26.634423 6156 catalog_manager.cc:5484] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.639158 14242 consensus_queue.cc:1035] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.642360 14351 consensus_queue.cc:1035] T 0df75a72a28349458c7cf145ad9b4715 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.652436 14243 consensus_queue.cc:1035] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.655193 13931 consensus_queue.cc:1035] T 6a5b794d73f54463a25c52b49f8dee71 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:26.672497 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.698587 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:26.703892 8559 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704046 8824 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704167 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.704154 6670 tablet_service.cc:1467] Processing CreateTablet for tablet fbb859e2ace341e7bb7406fae3e40ddc (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.704335 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.704383 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb859e2ace341e7bb7406fae3e40ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.706094 14302 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.706440 14303 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:26.706410 6408 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.706672 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.706997 14302 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.707257 14360 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:26.707294 14303 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.707984 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.708233 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.708464 14360 tablet_bootstrap.cc:654] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.708909 14363 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:26.709192 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.709466 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.709818 14363 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.709897 14302 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.709968 14302 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:26.710377 14302 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.710482 14302 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.710522 14302 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.710620 14302 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.707984 6279 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.710875 14302 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.710961 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.711449 14306 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:26.711766 14364 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:26.712198 14306 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.712600 14364 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.713320 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 60cea9bc061142bd9c8d2a6dec996e1a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.713371 8822 tablet_service.cc:1467] Processing CreateTablet for tablet fe516fb3caa44aafb16c762b6da49dcd (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=4ab34cd8a82949deb93ac207b72f9b11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:26.713559 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60cea9bc061142bd9c8d2a6dec996e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.714365 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe516fb3caa44aafb16c762b6da49dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:26.714815 14306 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:26.714890 14306 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.715286 14306 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.715379 14306 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.715409 14306 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.715515 14306 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.715773 14306 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.716949 14303 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:26.717031 14303 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:26.717460 14303 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.717558 14303 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.717587 14303 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.717695 14303 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.717788 14363 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:26.717862 14363 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:26.717940 14303 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:26.718688 14302 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.718935 14363 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.719026 14363 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.719056 14363 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.719188 14363 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.719434 14363 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:26.719514 14302 tablet_bootstrap.cc:654] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.720161 14360 tablet_bootstrap.cc:492] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:26.720244 14360 ts_tablet_manager.cc:1367] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:26.720690 14360 raft_consensus.cc:348] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.720837 14360 raft_consensus.cc:374] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.720901 14360 raft_consensus.cc:729] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.721046 14360 consensus_queue.cc:260] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.721345 14360 ts_tablet_manager.cc:1398] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.724042 14302 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.724175 14302 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:26.724342 14364 tablet_bootstrap.cc:492] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:26.724460 14364 ts_tablet_manager.cc:1367] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:26.724823 14302 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.724941 14302 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.724992 14302 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.725144 14302 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
W20240503 07:59:26.726137 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.726253 14364 raft_consensus.cc:348] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.726467 14302 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.726478 14364 raft_consensus.cc:374] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.726522 14364 raft_consensus.cc:729] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.726675 14302 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:26.726662 14364 consensus_queue.cc:260] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.727025 14364 ts_tablet_manager.cc:1398] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:26.727280 14364 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:26.728104 14364 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.728156 14302 tablet_bootstrap.cc:654] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:26.729203 14364 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:26.729285 14364 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:26.729683 14364 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.729801 14364 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.729864 14364 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.729992 14364 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.730091 14302 tablet_bootstrap.cc:492] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:26.730170 14302 ts_tablet_manager.cc:1367] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:26.730230 14364 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:26.730660 14302 raft_consensus.cc:348] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.731333 14302 raft_consensus.cc:374] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:26.731410 14302 raft_consensus.cc:729] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:26.731524 14302 consensus_queue.cc:260] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.731783 14302 ts_tablet_manager.cc:1398] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:26.744493 14243 raft_consensus.cc:482] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.744602 14243 raft_consensus.cc:504] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.745110 14243 leader_election.cc:288] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.745421 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.745404 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:26.745544 8843 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.745540 6298 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:26.745796 6496 leader_election.cc:302] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.745963 14243 raft_consensus.cc:2781] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.746019 14243 raft_consensus.cc:482] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.746064 14243 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.746843 14243 raft_consensus.cc:504] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.747190 14243 leader_election.cc:288] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.747469 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.747470 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60cea9bc061142bd9c8d2a6dec996e1a" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:26.747567 6298 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.747568 8843 raft_consensus.cc:3037] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.748417 6298 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.748452 8843 raft_consensus.cc:2454] T 60cea9bc061142bd9c8d2a6dec996e1a P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:26.748701 6496 leader_election.cc:302] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:26.748857 14243 raft_consensus.cc:2781] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.748944 14243 raft_consensus.cc:686] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:26.749063 14243 consensus_queue.cc:237] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.751052 6154 catalog_manager.cc:5484] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:26.753504 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 07:59:26.756299 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.756747 6560 raft_consensus.cc:1261] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.756822 6690 raft_consensus.cc:1261] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer 616fdda6ba714cb7be01a564de2b9db2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:26.757153 14242 consensus_queue.cc:1035] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.757297 14351 consensus_queue.cc:1035] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:26.760497 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.761656 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.761679 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.761713 14218 mvcc.cc:205] Tried to move back new op lower bound from 7023506091031883776 to 7023506090527420416. Current Snapshot: MvccSnapshot[applied={T|T < 7023506091031883776}]
I20240503 07:59:26.762799 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.763051 14377 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.763751 6156 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.763849 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:26.764164 14376 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.765264 6148 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314]). Expected version 1 got 0
I20240503 07:59:26.766072 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 1)
I20240503 07:59:26.764164 14375 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
W20240503 07:59:26.777298 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:26.787761 14289 raft_consensus.cc:482] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.787843 14289 raft_consensus.cc:504] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.788200 14289 leader_election.cc:288] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.788527 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.788532 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:26.788640 8581 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:26.788630 8843 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:26.788924 6625 leader_election.cc:302] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.789250 14197 raft_consensus.cc:2781] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.789314 14197 raft_consensus.cc:482] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.789391 14197 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.790138 14197 raft_consensus.cc:504] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.790413 14197 leader_election.cc:288] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:26.790673 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:26.790668 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb859e2ace341e7bb7406fae3e40ddc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.790782 8843 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.790781 8581 raft_consensus.cc:3037] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.791468 8843 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:26.791491 8581 raft_consensus.cc:2454] T fbb859e2ace341e7bb7406fae3e40ddc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:26.791761 6625 leader_election.cc:302] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:26.791908 14289 raft_consensus.cc:2781] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.791970 14289 raft_consensus.cc:686] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:26.792083 14289 consensus_queue.cc:237] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:26.794021 6148 catalog_manager.cc:5484] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:26.826154 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:26.842139 14287 raft_consensus.cc:482] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:26.842235 14287 raft_consensus.cc:504] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.842666 14287 leader_election.cc:288] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:26.842932 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:26.843005 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:26.843057 8842 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:26.843123 6298 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:26.843327 6365 leader_election.cc:302] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.843600 14287 raft_consensus.cc:2781] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:26.843662 14287 raft_consensus.cc:482] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:26.843699 14287 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.844504 14287 raft_consensus.cc:504] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.844825 14287 leader_election.cc:288] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:26.845094 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:26.845130 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe516fb3caa44aafb16c762b6da49dcd" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:26.845197 8842 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.845259 6298 raft_consensus.cc:3037] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:26.846105 6298 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:26.846112 8842 raft_consensus.cc:2454] T fe516fb3caa44aafb16c762b6da49dcd P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:26.846400 6365 leader_election.cc:302] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:26.846551 14287 raft_consensus.cc:2781] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:26.846632 14287 raft_consensus.cc:686] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:26.846820 14287 consensus_queue.cc:237] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:26.849140 6148 catalog_manager.cc:5484] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:26.869614 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:26.874861 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875108 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875200 14375 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:26.875391 14377 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:26.875627 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:26.875926 14376 tablet.cc:1620] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:26.880432 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 2)
W20240503 07:59:26.960742 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:26.962862 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:26.983690 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:26.991406 14375 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.992277 14376 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.992472 14377 tablet.cc:1722] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:26.998104 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314] alter complete (version 3)
I20240503 07:59:27.102984 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.103133 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.105767 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.106387 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:27.106422 6539 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:27.106671 8822 tablet_service.cc:1514] Processing DeleteTablet for tablet 374f4273dd194b3491e8542de68caba3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:27.106781 14381 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:27.106798 14380 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:27.106932 14382 tablet_replica.cc:330] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:27.106933 14381 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.106932 14380 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.107048 14381 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107048 14380 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107060 14382 raft_consensus.cc:2229] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:27.107250 14382 raft_consensus.cc:2258] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.107785 14382 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.107905 14380 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.107928 14381 ts_tablet_manager.cc:1875] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.109881 14382 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.109947 14382 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110059 14381 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.110121 14380 ts_tablet_manager.cc:1888] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.110172 14381 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110199 14380 log.cc:1196] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/374f4273dd194b3491e8542de68caba3
I20240503 07:59:27.110257 14382 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:27.110505 14381 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:27.110504 14380 ts_tablet_manager.cc:1909] T 374f4273dd194b3491e8542de68caba3 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:27.110849 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
I20240503 07:59:27.111040 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
I20240503 07:59:27.111076 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 374f4273dd194b3491e8542de68caba3 (table rebalancer_dynamic_table_new_name [id=cc15d983c8c14fe38b14e2ebe5a5d314]) successfully deleted
W20240503 07:59:27.117175 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.145772 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.204242 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.207432 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.212152 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212328 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212409 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212575 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.212622 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]), partition=
I20240503 07:59:27.212853 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f13edc131ea49c9bfd552f0581ea546. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.214320 14383 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.214679 14363 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:27.214815 14364 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:27.215545 14383 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.215546 14363 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.215641 14364 tablet_bootstrap.cc:654] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.216523 14364 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:27.216604 14364 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.216900 14383 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.216943 14363 tablet_bootstrap.cc:492] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:27.216979 14383 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.217013 14363 ts_tablet_manager.cc:1367] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.217075 14364 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217164 14364 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217193 14364 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217294 14364 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217387 14383 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217429 14363 raft_consensus.cc:348] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217541 14363 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217546 14364 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.217574 14363 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217466 14383 raft_consensus.cc:374] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.217621 14383 raft_consensus.cc:729] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.217676 14363 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217753 14383 consensus_queue.cc:260] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.217948 14363 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.218020 14383 ts_tablet_manager.cc:1398] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.284035 14197 consensus_queue.cc:1035] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.286871 14197 consensus_queue.cc:1035] T fbb859e2ace341e7bb7406fae3e40ddc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.313598 14243 consensus_queue.cc:1035] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.316380 13931 consensus_queue.cc:1035] T 60cea9bc061142bd9c8d2a6dec996e1a P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.340521 14335 raft_consensus.cc:482] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.340610 14335 raft_consensus.cc:504] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.341017 14335 leader_election.cc:288] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.341275 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:27.341431 6430 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:27.341691 6235 leader_election.cc:302] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:27.341720 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:27.341823 14335 raft_consensus.cc:2781] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.341873 14335 raft_consensus.cc:482] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.341907 14335 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.341889 8975 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:27.342895 14335 raft_consensus.cc:504] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.343209 14335 leader_election.cc:288] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.343484 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:27.343575 8975 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.343695 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f13edc131ea49c9bfd552f0581ea546" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:27.343798 6430 raft_consensus.cc:3037] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.344470 8975 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:27.344614 6430 raft_consensus.cc:2454] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:27.344797 6234 leader_election.cc:302] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:27.344950 14335 raft_consensus.cc:2781] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.345094 14335 raft_consensus.cc:686] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:27.345250 14335 consensus_queue.cc:237] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:27.346992 6156 catalog_manager.cc:5484] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:27.364709 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:27.370169 6670 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.370447 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.372280 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.372522 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.372676 14396 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.372807 8957 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.373024 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.373582 14396 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.374179 14383 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.375087 14383 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.375399 8822 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.375416 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.375695 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.375859 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.376157 6539 tablet_service.cc:1467] Processing CreateTablet for tablet b39cd44c3fe7460d89dc43ac0705f18b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.376444 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39cd44c3fe7460d89dc43ac0705f18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.376492 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.376689 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.378187 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d9816e0d19854fec917417815a21ba25 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.378410 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9816e0d19854fec917417815a21ba25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.378638 14400 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.378800 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 6a30765c934a4a69809cf03df6dcb110 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=6f18f7de4a7344139f213b113f23d5a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:27.379011 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a30765c934a4a69809cf03df6dcb110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.379527 14383 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.379602 14383 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:27.379649 14400 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.380057 14383 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.380147 14383 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.380177 14383 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.380281 14383 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.380563 14383 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:27.380729 14383 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:27.380833 14396 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.380905 14396 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:27.381299 14396 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.381449 14396 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.381510 14396 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.381573 14383 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.381928 14400 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.381915 14396 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.382043 14400 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.383354 14383 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:27.383435 14383 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.383466 14400 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.383559 14396 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.383566 14400 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.383598 14400 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.383688 14400 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.383744 14396 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.383984 14400 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:27.384013 14383 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.384101 14383 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.384132 14383 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.384142 14400 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.384244 14383 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.384500 14383 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.385038 14400 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.385922 14396 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.386540 14399 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:27.386852 14396 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.386950 14396 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:27.387423 14396 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.389143 14396 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.389219 14396 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.389348 14396 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.389604 14204 consensus_queue.cc:1035] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.389688 14396 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.389827 14399 tablet_bootstrap.cc:654] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.391691 14399 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:27.391798 14399 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:27.392221 14399 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.392390 14399 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.392449 14399 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.392598 14399 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.392937 14399 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.393085 14400 tablet_bootstrap.cc:492] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.393107 14399 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:27.393193 14400 ts_tablet_manager.cc:1367] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:27.393649 14400 raft_consensus.cc:348] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.393772 14400 raft_consensus.cc:374] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.393826 14400 raft_consensus.cc:729] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.393973 14400 consensus_queue.cc:260] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.394088 14399 tablet_bootstrap.cc:654] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.394447 14400 ts_tablet_manager.cc:1398] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.394703 14400 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:27.395735 14400 tablet_bootstrap.cc:654] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.397153 14400 tablet_bootstrap.cc:492] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:27.397234 14400 ts_tablet_manager.cc:1367] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:27.397730 14400 raft_consensus.cc:348] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.397812 14400 raft_consensus.cc:374] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.397840 14400 raft_consensus.cc:729] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.397953 14400 consensus_queue.cc:260] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.398204 14400 ts_tablet_manager.cc:1398] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.398849 14399 tablet_bootstrap.cc:492] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:27.398922 14399 ts_tablet_manager.cc:1367] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:27.399329 14399 raft_consensus.cc:348] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.399433 14399 raft_consensus.cc:374] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.399484 14399 raft_consensus.cc:729] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.399614 14399 consensus_queue.cc:260] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.399865 14399 ts_tablet_manager.cc:1398] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.400609 14287 consensus_queue.cc:1035] T fe516fb3caa44aafb16c762b6da49dcd P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:27.402832 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.440981 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:27.448565 13931 raft_consensus.cc:482] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.448680 13931 raft_consensus.cc:504] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.449102 13931 leader_election.cc:288] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.449662 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:27.449810 6690 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.450120 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:27.450273 8975 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.450264 6496 leader_election.cc:302] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.450436 13931 raft_consensus.cc:2781] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.450503 13931 raft_consensus.cc:482] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.450537 13931 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.451562 13931 raft_consensus.cc:504] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.451848 13931 leader_election.cc:288] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:27.452216 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:27.452307 8975 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.452307 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a30765c934a4a69809cf03df6dcb110" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:27.452407 6690 raft_consensus.cc:3037] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.453220 8975 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.453231 6690 raft_consensus.cc:2454] T 6a30765c934a4a69809cf03df6dcb110 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.453501 6496 leader_election.cc:302] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.453655 13931 raft_consensus.cc:2781] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.453742 13931 raft_consensus.cc:686] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:27.453892 13931 consensus_queue.cc:237] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:27.455924 6148 catalog_manager.cc:5484] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:27.465571 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:27.469017 8975 raft_consensus.cc:1261] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.469112 6430 raft_consensus.cc:1261] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:27.469429 14394 consensus_queue.cc:1035] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.469595 14335 consensus_queue.cc:1035] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.472429 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.472708 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
I20240503 07:59:27.472833 14416 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:27.473040 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.473415 14417 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:27.473783 6156 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
I20240503 07:59:27.474309 14392 mvcc.cc:205] Tried to move back new op lower bound from 7023506093949599744 to 7023506093447274496. Current Snapshot: MvccSnapshot[applied={T|T < 7023506093949599744}]
I20240503 07:59:27.478390 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:27.478796 14418 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:27.479373 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 1)
I20240503 07:59:27.479491 6154 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25]). Expected version 1 got 0
W20240503 07:59:27.519277 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.540104 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.557934 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.585453 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:27.590327 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.590970 14416 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:27.591161 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.591590 14418 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:27.591913 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:27.592190 14417 tablet.cc:1620] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:27.595327 13931 raft_consensus.cc:482] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.595428 13931 raft_consensus.cc:504] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.598647 13931 leader_election.cc:288] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.599016 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:27.599148 6690 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.599162 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 2)
I20240503 07:59:27.599503 6496 leader_election.cc:302] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.599673 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:27.599771 13931 raft_consensus.cc:2781] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.599797 8842 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:27.599823 13931 raft_consensus.cc:482] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.599859 13931 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.601137 13931 raft_consensus.cc:504] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.601670 13931 leader_election.cc:288] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.602034 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:27.602128 6690 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.604471 6690 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.604935 6496 leader_election.cc:302] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.605288 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39cd44c3fe7460d89dc43ac0705f18b" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:27.605388 8842 raft_consensus.cc:3037] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.606253 8842 raft_consensus.cc:2454] T b39cd44c3fe7460d89dc43ac0705f18b P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:27.606776 13931 raft_consensus.cc:2781] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.606796 14334 raft_consensus.cc:482] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:27.606885 14334 raft_consensus.cc:504] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.606948 13931 raft_consensus.cc:686] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:27.607077 13931 consensus_queue.cc:237] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:27.607213 14334 leader_election.cc:288] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.607601 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:27.607591 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:27.607702 6560 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:27.607702 8842 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:27.607960 8912 leader_election.cc:302] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:27.608152 14334 raft_consensus.cc:2781] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:27.608206 14334 raft_consensus.cc:482] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:27.608248 14334 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.609074 14334 raft_consensus.cc:504] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.609378 14334 leader_election.cc:288] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:27.609328 6154 catalog_manager.cc:5484] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:27.609658 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:27.609717 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9816e0d19854fec917417815a21ba25" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:27.609750 6560 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.609794 8842 raft_consensus.cc:3037] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:27.610567 8842 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:27.610567 6560 raft_consensus.cc:2454] T d9816e0d19854fec917417815a21ba25 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:27.610863 8910 leader_election.cc:302] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:27.611128 14334 raft_consensus.cc:2781] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:27.611267 14334 raft_consensus.cc:686] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:27.611407 14334 consensus_queue.cc:237] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:27.613997 6148 catalog_manager.cc:5484] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:27.629859 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.630478 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.662199 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:27.664083 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.674923 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.701126 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:27.706084 14416 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.706624 14418 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.706624 14417 tablet.cc:1722] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:27.710902 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25] alter complete (version 3)
W20240503 07:59:27.753296 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:27.806010 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.813761 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.813913 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:27.816037 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.816617 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:27.816933 14423 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:27.816915 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:27.817075 14423 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.817173 14424 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:27.817247 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f13edc131ea49c9bfd552f0581ea546 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:27 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:27.817322 14424 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:27.817507 14425 tablet_replica.cc:330] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:27.817562 14424 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.817591 14423 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.817663 14425 raft_consensus.cc:2229] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:27.817829 14425 raft_consensus.cc:2258] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:27.818109 14424 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.818353 14423 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.818571 14425 ts_tablet_manager.cc:1875] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:27.820676 14423 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820688 14425 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820698 14424 ts_tablet_manager.cc:1888] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:27.820760 14424 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.820807 14423 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.820816 14425 log.cc:1196] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9f13edc131ea49c9bfd552f0581ea546
I20240503 07:59:27.821090 14425 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:27.821197 14423 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:27.821247 14424 ts_tablet_manager.cc:1909] T 9f13edc131ea49c9bfd552f0581ea546 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:27.821740 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
I20240503 07:59:27.821820 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
I20240503 07:59:27.821995 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 9f13edc131ea49c9bfd552f0581ea546 (table rebalancer_dynamic_table_new_name [id=a915a3627fda4adca6dbfbf8d79ecd25]) successfully deleted
W20240503 07:59:27.852015 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 07:59:27.860857 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:27.893347 14243 consensus_queue.cc:1035] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:27.896653 14243 consensus_queue.cc:1035] T 6a30765c934a4a69809cf03df6dcb110 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:27.910600 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:27.918206 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:27.922830 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.922925 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.923086 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.923148 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.923125 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]), partition=
I20240503 07:59:27.923358 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ba94edffe84cd7af271bf9c5cbb46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:27.925422 14434 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:27.925505 14432 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:27.925642 14433 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:27.926492 14432 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927086 14434 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927687 14432 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:27.927819 14433 tablet_bootstrap.cc:654] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:27.927923 14432 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:27.928467 14432 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.928655 14432 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.928742 14432 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.928958 14432 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.929250 14432 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.929693 14433 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:27.929808 14434 tablet_bootstrap.cc:492] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:27.929884 14434 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:27.930295 14434 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.930419 14434 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.930471 14434 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.930593 14434 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.931107 14434 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:27.929777 14433 ts_tablet_manager.cc:1367] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:27.931560 14433 raft_consensus.cc:348] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.931655 14433 raft_consensus.cc:374] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:27.931691 14433 raft_consensus.cc:729] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:27.931805 14433 consensus_queue.cc:260] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:27.932829 14433 ts_tablet_manager.cc:1398] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:27.992044 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:28.041558 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.113791 13931 consensus_queue.cc:1035] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:28.114255 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 07:59:28.118261 13931 consensus_queue.cc:1035] T b39cd44c3fe7460d89dc43ac0705f18b P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.122133 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:28.128211 8957 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128485 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.128540 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128540 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.128868 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.129187 6408 tablet_service.cc:1467] Processing CreateTablet for tablet bc98e59704884c41a19e798e9c62972f (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.129429 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.130415 14442 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.130566 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.130774 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.131457 14442 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.131477 14433 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.131722 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc98e59704884c41a19e798e9c62972f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.132434 14433 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.132951 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.133217 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.133669 14433 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.133750 14433 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:28.134300 14433 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.134423 14433 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.134457 14433 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.134478 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.134579 14433 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.134682 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.134956 14434 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.134958 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae21b0dd499427cb9311430b7b16fb1 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.135205 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae21b0dd499427cb9311430b7b16fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.135399 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 0b751a528c4f43cfa60c0d133f08a78b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=28f965ce76d1458abfc814e48b86dfb8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.135608 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b751a528c4f43cfa60c0d133f08a78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.135900 14434 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.137445 14432 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:28.137882 14446 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:28.138384 14445 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:28.138407 14432 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.139256 14446 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.139991 14442 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.141602 14442 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:28.141969 14433 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 07:59:28.142060 14442 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.142161 14442 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.142246 14442 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.142395 14442 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.142704 14432 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:28.142781 14432 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.142987 14446 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.143077 14446 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.143170 14432 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143262 14432 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.143292 14432 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.143389 14432 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143657 14432 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.143653 14442 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.143808 14442 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.143877 14446 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.143985 14446 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.144019 14446 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.144155 14446 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.144539 14446 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.146103 14446 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
W20240503 07:59:28.147189 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.148267 14442 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.149899 14445 tablet_bootstrap.cc:654] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.150848 14440 consensus_queue.cc:1035] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.152442 14442 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.152535 14442 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:28.148324 14446 tablet_bootstrap.cc:654] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.152978 14442 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.153070 14442 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.153101 14442 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.153220 14442 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.153486 14442 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.153973 14446 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.154044 14446 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:28.154419 14446 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.154503 14446 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.154529 14446 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.154629 14446 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.154886 14446 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.157104 14434 tablet_bootstrap.cc:492] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.157191 14434 ts_tablet_manager.cc:1367] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:28.157594 14434 raft_consensus.cc:348] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.157685 14434 raft_consensus.cc:374] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.157709 14434 raft_consensus.cc:729] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.157802 14434 consensus_queue.cc:260] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.158053 14434 ts_tablet_manager.cc:1398] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.158200 14434 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.159014 14434 tablet_bootstrap.cc:654] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.159986 14334 consensus_queue.cc:1035] T d9816e0d19854fec917417815a21ba25 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.164151 14434 tablet_bootstrap.cc:492] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.164233 14434 ts_tablet_manager.cc:1367] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:28.164999 14434 raft_consensus.cc:348] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.165123 14434 raft_consensus.cc:374] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.165207 14434 raft_consensus.cc:729] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.165319 14434 consensus_queue.cc:260] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.165586 14434 ts_tablet_manager.cc:1398] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.169054 14445 tablet_bootstrap.cc:492] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:28.169137 14445 ts_tablet_manager.cc:1367] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 07:59:28.169782 14445 raft_consensus.cc:348] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.169884 14445 raft_consensus.cc:374] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.169957 14445 raft_consensus.cc:729] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.170094 14445 consensus_queue.cc:260] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.170372 14445 ts_tablet_manager.cc:1398] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.190119 14314 raft_consensus.cc:482] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.190182 14427 raft_consensus.cc:482] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.190209 14314 raft_consensus.cc:504] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.190265 14427 raft_consensus.cc:504] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.190563 14314 leader_election.cc:288] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.190636 14427 leader_election.cc:288] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:28.190821 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.190929 8581 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.190922 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.190914 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.191016 6296 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.191016 8975 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.191134 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:28.191177 6625 leader_election.cc:302] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.191244 6430 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.191256 8516 leader_election.cc:302] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.191313 14427 raft_consensus.cc:2781] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.191356 14427 raft_consensus.cc:482] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.191407 14427 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.191496 14314 raft_consensus.cc:2781] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.191545 14314 raft_consensus.cc:482] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.191592 14314 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.192332 14427 raft_consensus.cc:504] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.192461 14314 raft_consensus.cc:504] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.192658 14427 leader_election.cc:288] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:28.192790 14314 leader_election.cc:288] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.193066 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:28.193096 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ba94edffe84cd7af271bf9c5cbb46e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.193148 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.193183 8581 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193192 6430 raft_consensus.cc:3037] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193241 8975 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.193907 8581 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.194104 8975 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.194104 6430 raft_consensus.cc:2454] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.194149 6625 leader_election.cc:302] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.194288 14427 raft_consensus.cc:2781] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.194312 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b751a528c4f43cfa60c0d133f08a78b" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.194361 14427 raft_consensus.cc:686] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:28.194387 8516 leader_election.cc:302] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.194432 6296 raft_consensus.cc:3037] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.194528 14314 raft_consensus.cc:2781] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.194500 14427 consensus_queue.cc:237] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.194705 14314 raft_consensus.cc:686] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.194826 14314 consensus_queue.cc:237] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.195344 6296 raft_consensus.cc:2454] T 0b751a528c4f43cfa60c0d133f08a78b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.196837 6155 catalog_manager.cc:5484] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:28.197237 6148 catalog_manager.cc:5484] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.202980 14427 raft_consensus.cc:482] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.203058 14427 raft_consensus.cc:504] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.203379 14427 leader_election.cc:288] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.203611 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.203645 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.203718 6296 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.203738 8581 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:28.203949 6625 leader_election.cc:302] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.204103 14427 raft_consensus.cc:2781] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.204164 14427 raft_consensus.cc:482] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.204195 14427 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.204905 14427 raft_consensus.cc:504] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.205212 14427 leader_election.cc:288] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.205468 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.205564 6296 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.205569 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae21b0dd499427cb9311430b7b16fb1" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.205643 8581 raft_consensus.cc:3037] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.206360 8581 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.206360 6296 raft_consensus.cc:2454] T 6ae21b0dd499427cb9311430b7b16fb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:28.206673 6625 leader_election.cc:302] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.206950 14427 raft_consensus.cc:2781] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.207023 14427 raft_consensus.cc:686] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:28.207136 14427 consensus_queue.cc:237] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.209296 6148 catalog_manager.cc:5484] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.264884 14204 raft_consensus.cc:482] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.264977 14204 raft_consensus.cc:504] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.265416 14204 leader_election.cc:288] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.265657 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.265777 8975 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.265863 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:28.265990 8842 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.266036 6365 leader_election.cc:302] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.266312 14204 raft_consensus.cc:2781] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.266364 14204 raft_consensus.cc:482] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.266415 14204 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.267156 14204 raft_consensus.cc:504] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.267422 14204 leader_election.cc:288] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.267688 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.267727 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc98e59704884c41a19e798e9c62972f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:28.267791 8975 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.267819 8842 raft_consensus.cc:3037] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.268594 8842 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.268606 8975 raft_consensus.cc:2454] T bc98e59704884c41a19e798e9c62972f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.268851 6365 leader_election.cc:302] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.268994 14204 raft_consensus.cc:2781] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.269079 14204 raft_consensus.cc:686] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:28.269217 14204 consensus_queue.cc:237] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:28.271042 6148 catalog_manager.cc:5484] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:28.313283 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:28.317072 8581 raft_consensus.cc:1261] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.317193 6430 raft_consensus.cc:1261] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.317592 14426 consensus_queue.cc:1035] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.317970 14426 consensus_queue.cc:1035] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.320807 6155 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.321288 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.323559 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.324326 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.325902 14471 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.327626 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 1)
I20240503 07:59:28.328424 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.329134 14470 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.329989 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.330193 14466 mvcc.cc:205] Tried to move back new op lower bound from 7023506097423388672 to 7023506096926064640. Current Snapshot: MvccSnapshot[applied={T|T < 7023506097423388672}]
I20240503 07:59:28.330344 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.330651 14472 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:28.333117 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7]). Expected version 1 got 0
I20240503 07:59:28.433784 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:28.439113 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.439442 14470 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.439518 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.439790 14471 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.440182 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:28.440677 14472 tablet.cc:1620] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:28.444540 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 2)
I20240503 07:59:28.547570 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:28.552691 14470 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.553203 14472 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.553718 14471 tablet.cc:1722] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:28.558426 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7] alter complete (version 3)
W20240503 07:59:28.595721 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.616461 14460 consensus_queue.cc:1035] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.623111 14460 consensus_queue.cc:1035] T 0b751a528c4f43cfa60c0d133f08a78b P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.626332 14388 consensus_queue.cc:1035] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.629024 14427 consensus_queue.cc:1035] T 6ae21b0dd499427cb9311430b7b16fb1 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.660571 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.660718 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:28.662617 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.663144 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:28.663167 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:28.663256 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ba94edffe84cd7af271bf9c5cbb46e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:28 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:28.663468 14481 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:28.663522 14482 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:28.663596 14483 tablet_replica.cc:330] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:28.663630 14481 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.663636 14482 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:28.663736 14483 raft_consensus.cc:2229] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:28.663749 14482 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.663790 14481 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.664034 14483 raft_consensus.cc:2258] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:28.664371 14482 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.664371 14481 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.664553 14483 ts_tablet_manager.cc:1875] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:28.666708 14481 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666708 14482 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666734 14483 ts_tablet_manager.cc:1888] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:28.666780 14482 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.666800 14483 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.666922 14481 log.cc:1196] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/77ba94edffe84cd7af271bf9c5cbb46e
I20240503 07:59:28.667055 14482 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:28.667098 14483 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:28.667495 14481 ts_tablet_manager.cc:1909] T 77ba94edffe84cd7af271bf9c5cbb46e P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:28.667788 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
I20240503 07:59:28.667894 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
I20240503 07:59:28.668330 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 77ba94edffe84cd7af271bf9c5cbb46e (table rebalancer_dynamic_table_new_name [id=55ebb9bcc23c4e3b863070a0c3f84ed7]) successfully deleted
W20240503 07:59:28.759691 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.764277 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:28.769160 8559 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.769270 6540 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.769430 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.769497 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.771260 14487 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.771478 14486 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.771553 6670 tablet_service.cc:1467] Processing CreateTablet for tablet a7d390afc34147999069a4c2b13ef500 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]), partition=
I20240503 07:59:28.771771 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d390afc34147999069a4c2b13ef500. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 07:59:28.772208 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd4400894d14f839670d209468f1929: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:28.772429 14486 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.773561 14486 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.773648 14486 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.773942 14491 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:28.774044 14486 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.774150 14486 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.774183 14486 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.774291 14486 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.774576 14486 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.774945 14491 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.774945 14487 tablet_bootstrap.cc:654] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.776171 14487 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.776209 14491 tablet_bootstrap.cc:492] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:28.776289 14487 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:28.776372 14491 ts_tablet_manager.cc:1367] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.776803 14487 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.776935 14487 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.776986 14487 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.777045 14491 raft_consensus.cc:348] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777101 14487 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777145 14491 raft_consensus.cc:374] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.777189 14491 raft_consensus.cc:729] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.777302 14491 consensus_queue.cc:260] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.777380 14487 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.777597 14491 ts_tablet_manager.cc:1398] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:28.779444 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.798072 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:28.803248 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803372 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803539 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.803678 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.803701 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 46d537a71f6341e6979c62208a7eda90 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.803926 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d537a71f6341e6979c62208a7eda90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.805022 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.805197 14487 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.805269 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.805686 6277 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.805951 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.806169 14487 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.806535 8957 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.806676 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.806771 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.806914 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.808246 14495 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:28.808954 6540 tablet_service.cc:1467] Processing CreateTablet for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.809090 14486 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.809089 14496 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:28.809154 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f9882cc7cb4ea7978cfd9e56ac4dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.809828 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7b54f0ead1441d8ded1579e5e32aff (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=8180bac6b3fd4e8491f7b070d3b221da]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:28.809970 14496 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.810041 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7b54f0ead1441d8ded1579e5e32aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:28.810434 14486 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.812110 14486 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.812160 14487 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.812192 14486 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:28.812206 14495 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.812218 14487 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:28.812585 14487 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.812605 14486 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.812683 14486 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.812709 14486 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.812739 14487 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.812796 14487 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.812902 14487 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.813043 14486 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.813313 14486 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.813452 14486 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:28.813493 14497 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:28.813958 14487 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.814106 14487 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:28.814237 14486 tablet_bootstrap.cc:654] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.814417 14494 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.814915 14487 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815239 14494 tablet_bootstrap.cc:654] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815702 14497 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.815673 14204 consensus_queue.cc:1035] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.815856 14487 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:28.815927 14487 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.816289 14487 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.816377 14487 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.816408 14487 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.816517 14487 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.816605 14495 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:28.816668 14495 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:28.816751 14487 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.817034 14495 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.817114 14495 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.817138 14495 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.817230 14495 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.817453 14495 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:28.821820 14486 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:28.821895 14486 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:28.822100 14494 tablet_bootstrap.cc:492] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.822170 14494 ts_tablet_manager.cc:1367] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:28.822304 14486 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.822412 14486 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.822439 14486 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.822541 14486 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.822722 14494 raft_consensus.cc:348] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.822789 14486 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.822814 14494 raft_consensus.cc:374] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.822844 14494 raft_consensus.cc:729] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.824024 14494 consensus_queue.cc:260] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.824415 14494 ts_tablet_manager.cc:1398] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.824610 14494 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:28.825549 14494 tablet_bootstrap.cc:654] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:28.826421 14494 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:28.826543 14494 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:28.826939 14494 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.827864 14494 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.827913 14494 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.828032 14494 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.828279 14494 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:28.828799 14497 tablet_bootstrap.cc:492] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:28.828879 14497 ts_tablet_manager.cc:1367] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 07:59:28.828994 14496 tablet_bootstrap.cc:492] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:28.829078 14496 ts_tablet_manager.cc:1367] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 07:59:28.829275 14497 raft_consensus.cc:348] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.829375 14497 raft_consensus.cc:374] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.829402 14497 raft_consensus.cc:729] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.829515 14496 raft_consensus.cc:348] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.829619 14496 raft_consensus.cc:374] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:28.829653 14496 raft_consensus.cc:729] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:28.829754 14496 consensus_queue.cc:260] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.830006 14496 ts_tablet_manager.cc:1398] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:28.831568 14497 consensus_queue.cc:260] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.832430 14497 ts_tablet_manager.cc:1398] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:28.838491 14204 consensus_queue.cc:1035] T bc98e59704884c41a19e798e9c62972f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.844486 14475 raft_consensus.cc:482] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.844595 14475 raft_consensus.cc:504] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.845005 14475 leader_election.cc:288] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:28.845291 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:28.845430 6560 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.845500 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:28.845615 6690 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.845690 8514 leader_election.cc:302] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.845836 14475 raft_consensus.cc:2781] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.845886 14475 raft_consensus.cc:482] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.845928 14475 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.846892 14475 raft_consensus.cc:504] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.847208 14475 leader_election.cc:288] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:28.847462 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:28.847539 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d390afc34147999069a4c2b13ef500" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:28.847579 6690 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.847648 6560 raft_consensus.cc:3037] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.848284 6690 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.848502 6560 raft_consensus.cc:2454] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.848556 8516 leader_election.cc:302] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.848696 14475 raft_consensus.cc:2781] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.848817 14475 raft_consensus.cc:686] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.848935 14475 consensus_queue.cc:237] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:28.850958 6154 catalog_manager.cc:5484] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.898754 14204 raft_consensus.cc:482] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.898859 14204 raft_consensus.cc:504] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.899188 14204 leader_election.cc:288] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.899494 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:28.899629 6560 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.899601 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:28.899713 8575 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:28.899889 6363 leader_election.cc:302] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.900055 14204 raft_consensus.cc:2781] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.900110 14204 raft_consensus.cc:482] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.900136 14204 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.901054 14204 raft_consensus.cc:504] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.901403 14204 leader_election.cc:288] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:28.901666 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:28.901702 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d537a71f6341e6979c62208a7eda90" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:28.901773 6560 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.901801 8575 raft_consensus.cc:3037] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.902683 8575 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.902714 6560 raft_consensus.cc:2454] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:28.902916 6363 leader_election.cc:302] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.903079 14204 raft_consensus.cc:2781] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.903177 14204 raft_consensus.cc:686] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:28.903299 14204 consensus_queue.cc:237] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:28.905288 6154 catalog_manager.cc:5484] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:28.922616 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:28.937292 6346 tablet.cc:2369] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:28.937682 14243 raft_consensus.cc:482] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.937764 14243 raft_consensus.cc:504] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.938129 14243 leader_election.cc:288] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.938364 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:28.938477 6296 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:28.938719 6494 leader_election.cc:302] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:28.938755 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:28.938871 8976 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:28.938875 14243 raft_consensus.cc:2781] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.938926 14243 raft_consensus.cc:482] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.938958 14243 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.939818 14243 raft_consensus.cc:504] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.940091 14243 leader_election.cc:288] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:28.940423 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:28.940471 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f9882cc7cb4ea7978cfd9e56ac4dde" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:28.940528 6296 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.940569 8976 raft_consensus.cc:3037] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.941315 8976 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:28.941365 6296 raft_consensus.cc:2454] T e9f9882cc7cb4ea7978cfd9e56ac4dde P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:28.941572 6496 leader_election.cc:302] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:28.941722 14243 raft_consensus.cc:2781] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.941790 14243 raft_consensus.cc:686] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:28.941929 14243 consensus_queue.cc:237] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:28.943953 6154 catalog_manager.cc:5484] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:28.949069 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:28.961504 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:28.964726 6690 raft_consensus.cc:1261] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.965116 6560 raft_consensus.cc:1261] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:28.965174 14460 consensus_queue.cc:1035] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.965596 14460 consensus_queue.cc:1035] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:28.965880 14460 raft_consensus.cc:482] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:28.965955 14460 raft_consensus.cc:504] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.966392 14460 leader_election.cc:288] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.966730 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:28.966867 6429 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.967156 8517 leader_election.cc:302] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:28.967558 14475 raft_consensus.cc:2781] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:28.967614 14475 raft_consensus.cc:482] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:28.967650 14475 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.967979 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.968400 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:28.968516 8843 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:28.968576 14475 raft_consensus.cc:504] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.968719 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.968904 14475 leader_election.cc:288] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:28.969019 14471 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:28.970364 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.970772 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.970973 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:28.971074 14470 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:28.971283 14518 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:28.971657 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:28.971765 8843 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.972223 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.972678 8843 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.973014 8516 leader_election.cc:302] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:28.973176 14475 raft_consensus.cc:2781] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:28.973258 14475 raft_consensus.cc:686] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:28.973385 14475 consensus_queue.cc:237] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:28.973416 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7b54f0ead1441d8ded1579e5e32aff" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:28.973506 6429 raft_consensus.cc:3037] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:28.974417 6429 raft_consensus.cc:2454] T 4a7b54f0ead1441d8ded1579e5e32aff P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:28.974777 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10]). Expected version 1 got 0
I20240503 07:59:28.977245 6150 catalog_manager.cc:5484] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:28.978574 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 1)
W20240503 07:59:29.001771 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.029019 6608 tablet.cc:2369] T 46d537a71f6341e6979c62208a7eda90 P ad910f92ef9d4db8a8962cecba289f8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:29.036656 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.038589 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.082777 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.088330 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.088666 14471 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:29.088801 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.089097 14518 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.091609 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.091909 14470 tablet.cc:1620] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.094230 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 2)
W20240503 07:59:29.095244 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:29.106216 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:29.183944 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.196058 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:29.201210 14471 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.201762 14470 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:29.201819 14518 tablet.cc:1722] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:29.207600 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.208379 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10] alter complete (version 3)
W20240503 07:59:29.269780 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.308950 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.309079 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:29.311028 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.311560 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:29.311654 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:29.311839 14524 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:29.311965 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d390afc34147999069a4c2b13ef500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:29 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:29.312060 14524 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:29.312106 14525 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:29.312237 14526 tablet_replica.cc:330] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:29.312248 14524 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312274 14525 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.312389 14526 raft_consensus.cc:2229] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:29.312474 14525 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312553 14526 raft_consensus.cc:2258] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:29.312847 14524 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.313082 14526 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.313078 14525 ts_tablet_manager.cc:1875] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:29.315052 14524 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315124 14526 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315124 14525 ts_tablet_manager.cc:1888] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:29.315162 14524 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315196 14525 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315316 14526 log.cc:1196] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7d390afc34147999069a4c2b13ef500
I20240503 07:59:29.315411 14524 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:29.315497 14525 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:29.315630 14526 ts_tablet_manager.cc:1909] T a7d390afc34147999069a4c2b13ef500 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:29.316071 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.316234 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.316648 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet a7d390afc34147999069a4c2b13ef500 (table rebalancer_dynamic_table_new_name [id=bb309c41c11c440e8d6c9222e8888d10]) successfully deleted
I20240503 07:59:29.333252 14522 consensus_queue.cc:1035] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.336436 14508 consensus_queue.cc:1035] T 46d537a71f6341e6979c62208a7eda90 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.355346 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:29.412719 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:29.417976 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.418227 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.418378 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.418603 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.419986 14529 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:29.420238 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]), partition=
I20240503 07:59:29.420327 14530 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.420449 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad25e4f6ee24d409516cfe98cc11922. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.421403 14530 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.422390 14529 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.422613 14532 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.423478 14529 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:29.423501 14532 tablet_bootstrap.cc:654] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.423542 14529 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:29.423934 14529 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.424064 14529 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.424090 14529 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.424196 14529 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.424474 14529 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.425647 14532 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.425767 14532 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.426190 14532 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.426286 14532 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.426316 14532 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.426465 14532 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.430016 14532 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:29.433341 14530 tablet_bootstrap.cc:492] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.433437 14530 ts_tablet_manager.cc:1367] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:29.433835 14530 raft_consensus.cc:348] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.433938 14530 raft_consensus.cc:374] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.433974 14530 raft_consensus.cc:729] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.434103 14530 consensus_queue.cc:260] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.434371 14530 ts_tablet_manager.cc:1398] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.435289 14475 consensus_queue.cc:1035] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.439515 14517 consensus_queue.cc:1035] T 4a7b54f0ead1441d8ded1579e5e32aff P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.440456 6739 tablet.cc:2369] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:29.463778 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.465698 14243 consensus_queue.cc:1035] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.469139 14356 consensus_queue.cc:1035] T e9f9882cc7cb4ea7978cfd9e56ac4dde P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.515164 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:29.520664 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521065 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.521224 6672 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521461 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.521414 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.521667 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.523082 14532 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.523319 8822 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.523599 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.523947 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 038c4914d2b543ce9191051cf02225cd (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.524066 14532 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.524187 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038c4914d2b543ce9191051cf02225cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.525693 14532 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.525775 14532 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.526154 14544 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:29.526129 6278 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.526208 14532 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.526299 14532 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.526329 14532 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.526345 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.526454 6279 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.526449 14532 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.526628 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.526737 14532 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.527312 14544 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.528208 14532 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:29.528507 6408 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb7a38d75c49cf84fb56e3a6572508 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.528690 6410 tablet_service.cc:1467] Processing CreateTablet for tablet e74f3265657e4cbda4fe0b02e6d148db (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=3393a1c12b3d46ecb448c3c3cdbb6c3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:29.528759 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb7a38d75c49cf84fb56e3a6572508. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.528941 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74f3265657e4cbda4fe0b02e6d148db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:29.530683 14530 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.531543 14530 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.531769 14542 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:29.533136 14542 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.534217 14542 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:29.534350 14542 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:29.534488 14532 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.534718 14542 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.534827 14542 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.534858 14542 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.534986 14542 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.535331 14542 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.535431 14532 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:29.535511 14532 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:29.535825 14544 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:29.535868 14532 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.535950 14532 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.535981 14532 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.536011 14544 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:29.536309 14532 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.536517 14544 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.536638 14544 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.536667 14544 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.536757 14544 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.535097 14545 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.536898 14532 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.537184 14544 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.537782 14545 tablet_bootstrap.cc:654] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.539252 14530 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.539326 14530 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:29.539731 14530 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.539827 14530 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.539868 14530 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.539966 14530 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.539311 14545 tablet_bootstrap.cc:492] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.540227 14530 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.540406 14530 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:29.541419 14530 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.542665 14545 ts_tablet_manager.cc:1367] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 07:59:29.543141 14545 raft_consensus.cc:348] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.543248 14545 raft_consensus.cc:374] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.543279 14545 raft_consensus.cc:729] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.543388 14545 consensus_queue.cc:260] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.543648 14545 ts_tablet_manager.cc:1398] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.543879 14545 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.544760 14530 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:29.544873 14530 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 07:59:29.544982 14545 tablet_bootstrap.cc:654] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.545303 14530 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.545395 14530 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.545428 14530 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.545543 14530 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.545763 14530 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:29.546083 14545 tablet_bootstrap.cc:492] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.546172 14545 ts_tablet_manager.cc:1367] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:29.546603 14545 raft_consensus.cc:348] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.546685 14545 raft_consensus.cc:374] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.546715 14545 raft_consensus.cc:729] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.546969 14545 consensus_queue.cc:260] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.547362 14545 ts_tablet_manager.cc:1398] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:29.547546 14545 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:29.548377 14545 tablet_bootstrap.cc:654] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:29.549391 14545 tablet_bootstrap.cc:492] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:29.549494 14545 ts_tablet_manager.cc:1367] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:29.549906 14545 raft_consensus.cc:348] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.550009 14545 raft_consensus.cc:374] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:29.550086 14545 raft_consensus.cc:729] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:29.550244 14545 consensus_queue.cc:260] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.550555 14545 ts_tablet_manager.cc:1398] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:29.555344 14523 raft_consensus.cc:482] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.555429 14523 raft_consensus.cc:504] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.555825 14523 leader_election.cc:288] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:29.556072 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:29.556144 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:29.556174 6296 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.556258 8575 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.556424 6625 leader_election.cc:302] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:29.556572 14523 raft_consensus.cc:2781] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.556628 14523 raft_consensus.cc:482] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.556664 14523 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.557509 14523 raft_consensus.cc:504] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.557794 14523 leader_election.cc:288] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:29.558038 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:29.558121 6296 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.558117 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038c4914d2b543ce9191051cf02225cd" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:29.558218 8575 raft_consensus.cc:3037] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.559010 8575 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.559068 6296 raft_consensus.cc:2454] T 038c4914d2b543ce9191051cf02225cd P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.559314 6625 leader_election.cc:302] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:29.559463 14523 raft_consensus.cc:2781] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.559546 14523 raft_consensus.cc:686] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:29.559679 14523 consensus_queue.cc:237] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:29.561522 6150 catalog_manager.cc:5484] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:29.587332 14335 raft_consensus.cc:482] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.587424 14335 raft_consensus.cc:504] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.587811 14335 leader_election.cc:288] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:29.588079 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:29.588099 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:29.588207 6690 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:29.588207 6429 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:29.588461 6235 leader_election.cc:302] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.588616 14335 raft_consensus.cc:2781] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.588662 14335 raft_consensus.cc:482] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.588696 14335 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.589491 14335 raft_consensus.cc:504] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.589787 14335 leader_election.cc:288] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:29.590049 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:29.590040 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb7a38d75c49cf84fb56e3a6572508" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:29.590145 6429 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.590145 6690 raft_consensus.cc:3037] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.591006 6690 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:29.591006 6429 raft_consensus.cc:2454] T d2bb7a38d75c49cf84fb56e3a6572508 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:29.591266 6235 leader_election.cc:302] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.591429 14335 raft_consensus.cc:2781] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.591508 14335 raft_consensus.cc:686] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:29.591640 14335 consensus_queue.cc:237] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:29.593873 6150 catalog_manager.cc:5484] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:29.616708 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.625414 14523 raft_consensus.cc:482] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.625504 14523 raft_consensus.cc:504] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.625881 14523 leader_election.cc:288] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:29.626121 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:29.626160 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:29.626250 6560 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.626266 6429 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:29.626557 6625 leader_election.cc:302] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:29.626680 14523 raft_consensus.cc:2781] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.626731 14523 raft_consensus.cc:482] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.626773 14523 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.627624 14523 raft_consensus.cc:504] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.627933 14523 leader_election.cc:288] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:29.628176 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:29.628253 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad25e4f6ee24d409516cfe98cc11922" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:29.628278 6560 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.628347 6429 raft_consensus.cc:3037] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.629191 6429 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.629220 6560 raft_consensus.cc:2454] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:29.629520 6625 leader_election.cc:302] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:29.630009 14523 raft_consensus.cc:2781] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.630088 14523 raft_consensus.cc:686] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:29.630203 14523 consensus_queue.cc:237] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
W20240503 07:59:29.631373 6477 tablet.cc:2369] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:29.632272 6150 catalog_manager.cc:5484] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:29.692864 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:29.754390 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.764444 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:29.768700 6429 raft_consensus.cc:1261] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.768700 6560 raft_consensus.cc:1261] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:29.769065 14427 consensus_queue.cc:1035] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.769201 14523 consensus_queue.cc:1035] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.771255 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.774173 6156 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.774224 14508 raft_consensus.cc:482] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:29.774296 14508 raft_consensus.cc:504] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.774178 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.774569 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.774632 14508 leader_election.cc:288] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:29.774847 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.774899 14565 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.774987 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:29.775107 8842 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:29.775215 6148 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.775383 6365 leader_election.cc:302] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.775771 14508 raft_consensus.cc:2781] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:29.775820 14508 raft_consensus.cc:482] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:29.775862 14508 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.775925 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:29.776039 6296 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:29.777153 14508 raft_consensus.cc:504] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.777390 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.777468 14508 leader_election.cc:288] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:29.777550 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:29.777712 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f]). Expected version 1 got 0
I20240503 07:59:29.777815 14563 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:29.778097 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:29.778205 8842 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.779255 8842 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:29.779680 6365 leader_election.cc:302] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:29.779981 14508 raft_consensus.cc:2781] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:29.780071 14508 raft_consensus.cc:686] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:29.780105 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74f3265657e4cbda4fe0b02e6d148db" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:29.780205 6296 raft_consensus.cc:3037] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:29.780207 14508 consensus_queue.cc:237] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:29.781136 6296 raft_consensus.cc:2454] T e74f3265657e4cbda4fe0b02e6d148db P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:29.782219 14562 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.782629 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 1)
I20240503 07:59:29.785281 6148 catalog_manager.cc:5484] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:29.887091 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:29.891726 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.892041 14562 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:29.892289 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.892678 14563 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:29.893712 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:29.894003 14565 tablet.cc:1620] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:29.896818 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 2)
W20240503 07:59:29.937093 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:29.939532 14427 consensus_queue.cc:1035] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:29.942755 14564 consensus_queue.cc:1035] T 038c4914d2b543ce9191051cf02225cd P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:29.981283 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 07:59:29.981863 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:29.998322 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.003535 14562 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.004472 14563 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:30.004536 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.009733 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f] alter complete (version 3)
W20240503 07:59:30.010143 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.010526 14565 tablet.cc:1722] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:30.025977 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 07:59:30.027217 14335 consensus_queue.cc:1035] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.030653 14335 consensus_queue.cc:1035] T d2bb7a38d75c49cf84fb56e3a6572508 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:30.081787 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.109874 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 07:59:30.113297 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.113441 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.115357 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.115940 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:30.115974 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:30.116108 6410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad25e4f6ee24d409516cfe98cc11922 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:30.116519 14577 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:30.116542 14576 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:30.116678 14576 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.116706 14577 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.116843 14576 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.116958 14577 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.117105 14575 tablet_replica.cc:330] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:30.117254 14575 raft_consensus.cc:2229] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.117430 14575 raft_consensus.cc:2258] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.117550 14577 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.117551 14576 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.117885 14575 ts_tablet_manager.cc:1875] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.119773 14575 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119836 14575 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.119835 14577 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119832 14576 ts_tablet_manager.cc:1888] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.119902 14576 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.119904 14577 log.cc:1196] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ad25e4f6ee24d409516cfe98cc11922
I20240503 07:59:30.120240 14577 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:30.120431 14575 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:30.120543 14576 ts_tablet_manager.cc:1909] T 1ad25e4f6ee24d409516cfe98cc11922 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:30.121068 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
I20240503 07:59:30.121098 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
I20240503 07:59:30.121250 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 1ad25e4f6ee24d409516cfe98cc11922 (table rebalancer_dynamic_table_new_name [id=7f5ed08151694cb1a89ca7924ac5539f]) successfully deleted
W20240503 07:59:30.122293 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 07:59:30.123034 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.138583 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 07:59:30.139135 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.187517 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 07:59:30.206079 14567 consensus_queue.cc:1035] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.208693 14204 consensus_queue.cc:1035] T e74f3265657e4cbda4fe0b02e6d148db P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.217085 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.221664 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.221841 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.221935 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.222061 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.224241 14583 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:30.224367 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe07fce94584c3b86d19c04588976eb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]), partition=
I20240503 07:59:30.224520 14582 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:30.224596 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe07fce94584c3b86d19c04588976eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.225442 14583 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.225488 14582 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.226857 14582 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:30.226971 14582 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.227105 14586 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.228008 14586 tablet_bootstrap.cc:654] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.228302 14583 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:30.228436 14583 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.229035 14583 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.229192 14583 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.229267 14583 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.229436 14583 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.229761 14583 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.230592 14582 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.230710 14582 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.230752 14582 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.230873 14582 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.231232 14582 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:30.232271 14586 tablet_bootstrap.cc:492] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.232347 14586 ts_tablet_manager.cc:1367] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:30.232656 14586 raft_consensus.cc:348] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.232749 14586 raft_consensus.cc:374] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.232841 14586 raft_consensus.cc:729] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.232960 14586 consensus_queue.cc:260] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.233227 14586 ts_tablet_manager.cc:1398] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 07:59:30.234864 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.266206 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.283321 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.322376 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:30.328003 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.328290 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.328477 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.328723 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.328963 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.329208 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.331048 14588 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:30.332090 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.332113 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.332329 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.332329 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.333309 8823 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.333571 8823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.334422 14582 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:30.334527 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 323e04514862427b9484b7feb53584c7 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.334743 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323e04514862427b9484b7feb53584c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.335281 14582 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.335517 14589 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:30.333303 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 60222ff0a6184874aaf599500254fa5e (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.335887 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60222ff0a6184874aaf599500254fa5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.336844 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 253d36eba03243c39dfbb4ca750383c9 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=db5ae078b6ce47cca23fb8aa76b0c75c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:30.336943 14589 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.337103 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253d36eba03243c39dfbb4ca750383c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.337947 14589 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:30.338021 14589 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:30.338399 14589 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.338495 14589 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.338522 14589 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.338620 14589 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.338876 14589 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.339082 14586 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.339191 14588 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.339850 14586 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.340219 14588 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:30.340289 14588 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:30.340698 14588 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.340837 14588 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.340893 14588 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.341022 14588 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.341346 14588 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.341506 14588 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:30.342072 14590 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:30.342414 14588 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.343155 14590 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.343708 14582 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:30.343788 14582 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:30.343992 14586 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.344066 14586 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:30.344197 14582 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.344286 14582 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344316 14582 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.344372 14590 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:30.344439 14590 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:30.344431 14582 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.344504 14586 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344594 14586 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344623 14586 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.344695 14582 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.344722 14586 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344863 14590 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.344986 14590 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.344987 14586 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.345041 14590 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.345177 14590 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.345544 14590 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.346360 14591 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.347333 14591 tablet_bootstrap.cc:654] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.348553 14588 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:30.348627 14588 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:30.349017 14591 tablet_bootstrap.cc:492] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.349076 14588 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.349148 14591 ts_tablet_manager.cc:1367] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:30.349161 14588 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.349189 14588 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.349292 14588 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.349557 14588 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.349623 14591 raft_consensus.cc:348] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.349709 14591 raft_consensus.cc:374] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.349754 14591 raft_consensus.cc:729] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.349869 14591 consensus_queue.cc:260] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.350211 14591 ts_tablet_manager.cc:1398] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.350467 14591 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.352270 14591 tablet_bootstrap.cc:654] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.354588 14335 raft_consensus.cc:482] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.354686 14335 raft_consensus.cc:504] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.355136 14335 leader_election.cc:288] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.355157 14591 tablet_bootstrap.cc:492] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.355226 14591 ts_tablet_manager.cc:1367] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:30.355406 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:30.355473 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:30.355511 8575 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:30.355579 8976 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:30.355638 14591 raft_consensus.cc:348] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.355751 14591 raft_consensus.cc:374] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.355791 14591 raft_consensus.cc:729] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.355820 6232 leader_election.cc:302] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:30.355921 14591 consensus_queue.cc:260] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.356128 14335 raft_consensus.cc:2781] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.356181 14335 raft_consensus.cc:482] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.356232 14335 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.356492 14591 ts_tablet_manager.cc:1398] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:30.356628 14591 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:30.357123 14335 raft_consensus.cc:504] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.357419 14335 leader_election.cc:288] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.357688 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:30.357803 8575 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.357950 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe07fce94584c3b86d19c04588976eb" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:30.358034 8976 raft_consensus.cc:3037] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.358405 14591 tablet_bootstrap.cc:654] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.358687 8575 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:30.358834 8976 raft_consensus.cc:2454] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:30.359088 6232 leader_election.cc:302] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:30.359580 14473 raft_consensus.cc:2781] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.359668 14473 raft_consensus.cc:686] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:30.359777 14473 consensus_queue.cc:237] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:30.360092 14591 tablet_bootstrap.cc:492] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:30.360306 14591 ts_tablet_manager.cc:1367] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
W20240503 07:59:30.360837 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 07:59:30.361799 14591 raft_consensus.cc:348] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.361771 6154 catalog_manager.cc:5484] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.361876 14591 raft_consensus.cc:374] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.361902 14591 raft_consensus.cc:729] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.362007 14591 consensus_queue.cc:260] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.362263 14591 ts_tablet_manager.cc:1398] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:30.379738 14564 raft_consensus.cc:482] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.379837 14564 raft_consensus.cc:504] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.380209 14564 leader_election.cc:288] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:30.380460 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:30.380488 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:30.380561 8842 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:30.380587 6429 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:30.380808 6627 leader_election.cc:302] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:30.380982 14564 raft_consensus.cc:2781] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.381037 14564 raft_consensus.cc:482] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.381062 14564 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.381906 14564 raft_consensus.cc:504] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.382230 14564 leader_election.cc:288] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:30.382498 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:30.382508 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60222ff0a6184874aaf599500254fa5e" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:30.382596 6429 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.382618 8842 raft_consensus.cc:3037] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.383440 8842 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:30.383440 6429 raft_consensus.cc:2454] T 60222ff0a6184874aaf599500254fa5e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:30.383678 6627 leader_election.cc:302] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:30.383829 14564 raft_consensus.cc:2781] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.383915 14564 raft_consensus.cc:686] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:30.384124 14564 consensus_queue.cc:237] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.386427 6156 catalog_manager.cc:5484] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:30.418143 14455 raft_consensus.cc:482] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.418231 14455 raft_consensus.cc:504] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.418628 14455 leader_election.cc:288] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:30.418886 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:30.418903 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:30.418989 6429 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:30.418987 8843 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:30.419220 8913 leader_election.cc:302] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:30.419380 14455 raft_consensus.cc:2781] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.419431 14455 raft_consensus.cc:482] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.419456 14455 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.420321 14455 raft_consensus.cc:504] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.420712 14455 leader_election.cc:288] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:30.420970 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:30.421077 8843 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.421142 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323e04514862427b9484b7feb53584c7" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:30.421243 6430 raft_consensus.cc:3037] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.422044 8843 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:30.422165 6430 raft_consensus.cc:2454] T 323e04514862427b9484b7feb53584c7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:30.422302 8912 leader_election.cc:302] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:30.422521 14455 raft_consensus.cc:2781] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.422629 14455 raft_consensus.cc:686] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:30.422766 14455 consensus_queue.cc:237] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:30.425107 6156 catalog_manager.cc:5484] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:30.438886 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.447893 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:30.454813 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.468209 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 07:59:30.477031 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:30.480540 8575 raft_consensus.cc:1261] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.480819 8976 raft_consensus.cc:1261] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:30.480916 14335 consensus_queue.cc:1035] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.481175 14601 consensus_queue.cc:1035] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.483469 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.483790 14610 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:30.484802 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.485088 14611 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:30.485658 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.485996 6154 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.486358 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
I20240503 07:59:30.489797 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:30.490132 14612 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:30.490492 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 1)
I20240503 07:59:30.493191 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016]). Expected version 1 got 0
W20240503 07:59:30.517398 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 07:59:30.518013 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:30.536352 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 07:59:30.577417 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:30.587909 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.595669 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:30.601086 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.601576 14610 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:30.601985 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.602355 14611 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:30.607026 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:30.607285 14612 tablet.cc:1620] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:30.607753 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 2)
W20240503 07:59:30.639019 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:30.671947 14291 raft_consensus.cc:482] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:30.672053 14291 raft_consensus.cc:504] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.672452 14291 leader_election.cc:288] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.672710 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:30.672739 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:30.672871 6560 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:30.672889 8575 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:30.673156 8776 leader_election.cc:302] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:30.673326 14291 raft_consensus.cc:2781] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:30.673380 14291 raft_consensus.cc:482] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:30.673424 14291 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.674185 14291 raft_consensus.cc:504] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.674456 14291 leader_election.cc:288] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:30.674739 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:30.674759 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253d36eba03243c39dfbb4ca750383c9" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:30.674834 6560 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.674844 8575 raft_consensus.cc:3037] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:30.675530 6560 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:30.675540 8575 raft_consensus.cc:2454] T 253d36eba03243c39dfbb4ca750383c9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:30.675772 8776 leader_election.cc:302] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:30.675913 14291 raft_consensus.cc:2781] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:30.675998 14291 raft_consensus.cc:686] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:30.676124 14291 consensus_queue.cc:237] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:30.678241 6148 catalog_manager.cc:5484] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:30.708830 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:30.713019 14610 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.713516 14612 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.713666 14611 tablet.cc:1722] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:30.717916 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016] alter complete (version 3)
I20240503 07:59:30.797247 14427 consensus_queue.cc:1035] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.799883 14523 consensus_queue.cc:1035] T 60222ff0a6184874aaf599500254fa5e P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.820238 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.820389 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:30.822582 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.823053 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:30.823215 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:30.823354 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fe07fce94584c3b86d19c04588976eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:30 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:30.823557 14622 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:30.823662 14624 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:30.823661 14623 tablet_replica.cc:330] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:30.823787 14622 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.823801 14623 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:30.823868 14624 raft_consensus.cc:2229] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:30.823984 14623 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824023 14624 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824066 14622 raft_consensus.cc:2258] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:30.824684 14624 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.824903 14622 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.824985 14623 ts_tablet_manager.cc:1875] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:30.826877 14623 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.826987 14623 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.827263 14622 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.827414 14622 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.827723 14622 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:30.828711 14624 ts_tablet_manager.cc:1888] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:30.828851 14624 log.cc:1196] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/2fe07fce94584c3b86d19c04588976eb
I20240503 07:59:30.828904 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.829221 14623 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:30.829258 14624 ts_tablet_manager.cc:1909] T 2fe07fce94584c3b86d19c04588976eb P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:30.829799 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.830472 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 2fe07fce94584c3b86d19c04588976eb (table rebalancer_dynamic_table_new_name [id=40c8af0a6d494e96b7cc08b3d1a9e016]) successfully deleted
I20240503 07:59:30.924307 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:30.929142 8560 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.929395 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.929436 6540 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.929675 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.931239 14625 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:30.931515 6670 tablet_service.cc:1467] Processing CreateTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]), partition=
I20240503 07:59:30.931588 14626 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:30.931735 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5511ac79f64b7ab3edbdafe8df3ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:30.932348 14625 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.932425 14626 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.933601 14625 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:30.933712 14625 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 07:59:30.934327 14629 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:30.934427 14626 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:30.934520 14626 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:30.934438 14625 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.934702 14625 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.934758 14625 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.934914 14625 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.934971 14626 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.935057 14626 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.935089 14626 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.935205 14629 tablet_bootstrap.cc:654] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:30.935209 14626 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.935488 14626 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:30.936545 14625 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:30.937696 14629 tablet_bootstrap.cc:492] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:30.937778 14629 ts_tablet_manager.cc:1367] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:30.938189 14629 raft_consensus.cc:348] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.938421 14629 raft_consensus.cc:374] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:30.939535 14629 raft_consensus.cc:729] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:30.939700 14629 consensus_queue.cc:260] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:30.940055 14629 ts_tablet_manager.cc:1398] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:30.968590 14455 consensus_queue.cc:1035] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:30.972138 14632 consensus_queue.cc:1035] T 323e04514862427b9484b7feb53584c7 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:30.999528 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:31.082301 14291 consensus_queue.cc:1035] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.086320 14291 consensus_queue.cc:1035] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.142741 14523 raft_consensus.cc:482] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.142832 14523 raft_consensus.cc:504] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.143141 14523 leader_election.cc:288] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.143429 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:31.143486 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.143582 8575 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.143594 6560 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.143839 6625 leader_election.cc:302] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.143997 14523 raft_consensus.cc:2781] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.144063 14523 raft_consensus.cc:482] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.144106 14523 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.145028 14523 raft_consensus.cc:504] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.145398 14523 leader_election.cc:288] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.145709 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:31.145825 8575 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.146009 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5511ac79f64b7ab3edbdafe8df3ad9" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.146123 6560 raft_consensus.cc:3037] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.146651 8575 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.146907 6560 raft_consensus.cc:2454] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.146915 6625 leader_election.cc:302] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.147073 14523 raft_consensus.cc:2781] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.147174 14523 raft_consensus.cc:686] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:31.147293 14523 consensus_queue.cc:237] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.149464 6156 catalog_manager.cc:5484] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:31.206988 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:31.212352 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.212478 8957 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.212652 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.212741 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.213524 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.213738 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.214720 14640 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.214946 8955 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.215142 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.215193 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 054c5cb68f4b47809b2c0dcfc3281d95 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.215451 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054c5cb68f4b47809b2c0dcfc3281d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.216477 8822 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.216713 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.217123 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.217195 14641 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.217337 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.218077 14641 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.219203 14640 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.219257 14644 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:31.219799 14629 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:31.220301 14640 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.220391 14640 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:31.220645 6540 tablet_service.cc:1467] Processing CreateTablet for tablet bc77e81660b44b6a9b1ae4f98cfaeab1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.220765 14629 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.220906 14642 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:31.220942 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc77e81660b44b6a9b1ae4f98cfaeab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.221091 14640 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.221197 14640 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.221230 14640 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.221349 14640 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.221644 14640 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.221719 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 702e72e3876a453a83ea779d4f5239d7 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=e7bd366c307747e1ba666618e1e7e9d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:31.221840 14640 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.221933 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702e72e3876a453a83ea779d4f5239d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.222262 14641 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.222353 14641 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:31.222757 14640 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.222788 14641 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.222887 14641 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.222923 14641 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.223155 14626 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:31.223367 14641 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.223680 14641 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.224035 14626 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.224078 14641 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.225347 14641 tablet_bootstrap.cc:654] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.225916 14644 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.227191 14626 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:31.227279 14626 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:31.227329 14640 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.227398 14640 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:31.227731 14626 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.227833 14626 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.227869 14626 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.227988 14626 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.228070 14642 tablet_bootstrap.cc:654] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.228291 14626 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.228904 14640 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229074 14629 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:31.229112 14640 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.229156 14629 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:31.229158 14640 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.229275 14640 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229519 14642 tablet_bootstrap.cc:492] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:31.229590 14642 ts_tablet_manager.cc:1367] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:31.229584 14629 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.229672 14629 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.229702 14629 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.229815 14629 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.230062 14629 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.230027 14642 raft_consensus.cc:348] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.230114 14642 raft_consensus.cc:374] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.230144 14642 raft_consensus.cc:729] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.230257 14642 consensus_queue.cc:260] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.230513 14642 ts_tablet_manager.cc:1398] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.231830 14640 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:31.232013 14640 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:31.233608 14640 tablet_bootstrap.cc:654] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.234637 14640 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:31.234722 14640 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:31.235240 14640 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.235332 14640 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.235360 14640 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.235462 14640 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.235731 14640 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.239686 14644 tablet_bootstrap.cc:492] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:31.239759 14644 ts_tablet_manager.cc:1367] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:31.239902 14641 tablet_bootstrap.cc:492] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.239970 14641 ts_tablet_manager.cc:1367] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240503 07:59:31.240281 14644 raft_consensus.cc:348] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.240293 14641 raft_consensus.cc:348] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.240370 14644 raft_consensus.cc:374] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.240386 14641 raft_consensus.cc:374] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.240398 14644 raft_consensus.cc:729] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.240409 14641 raft_consensus.cc:729] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.240514 14641 consensus_queue.cc:260] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.240552 14644 consensus_queue.cc:260] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.240746 14641 ts_tablet_manager.cc:1398] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.240890 14644 ts_tablet_manager.cc:1398] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.261148 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:31.264132 8575 raft_consensus.cc:1261] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.264138 6560 raft_consensus.cc:1261] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.264506 14427 consensus_queue.cc:1035] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.264624 14523 consensus_queue.cc:1035] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.267282 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.267427 14618 mvcc.cc:205] Tried to move back new op lower bound from 7023506109494931456 to 7023506109020590080. Current Snapshot: MvccSnapshot[applied={T|T < 7023506109494931456}]
I20240503 07:59:31.267577 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.267869 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.268023 14656 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:31.268818 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.269073 14657 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.271402 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 1)
I20240503 07:59:31.271597 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.271757 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0]). Expected version 1 got 0
I20240503 07:59:31.271869 14658 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.274926 14632 raft_consensus.cc:482] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.275012 14632 raft_consensus.cc:504] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.275409 14632 leader_election.cc:288] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:31.275655 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.275734 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:31.275777 6298 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:31.275861 6429 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:31.276054 8910 leader_election.cc:302] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:31.276261 14632 raft_consensus.cc:2781] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.276316 14632 raft_consensus.cc:482] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.276360 14632 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.277164 14632 raft_consensus.cc:504] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.277431 14632 leader_election.cc:288] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:31.277690 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.277722 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054c5cb68f4b47809b2c0dcfc3281d95" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:31.277776 6298 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.277803 6429 raft_consensus.cc:3037] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.278425 6298 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:31.278436 6429 raft_consensus.cc:2454] T 054c5cb68f4b47809b2c0dcfc3281d95 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:31.278683 8913 leader_election.cc:302] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:31.278836 14632 raft_consensus.cc:2781] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.278909 14632 raft_consensus.cc:686] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:31.279022 14632 consensus_queue.cc:237] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:31.281167 6150 catalog_manager.cc:5484] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.298610 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.319641 14636 raft_consensus.cc:482] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.319736 14636 raft_consensus.cc:504] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.320152 14636 leader_election.cc:288] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.320420 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:31.320473 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.320533 8975 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:31.320593 6560 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:31.320865 8776 leader_election.cc:302] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.321028 14636 raft_consensus.cc:2781] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.321077 14636 raft_consensus.cc:482] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.321111 14636 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.322000 14636 raft_consensus.cc:504] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.322274 14636 leader_election.cc:288] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.322559 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:31.322558 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc77e81660b44b6a9b1ae4f98cfaeab1" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.322659 8975 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.322674 6560 raft_consensus.cc:3037] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.323468 8975 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:31.323575 6560 raft_consensus.cc:2454] T bc77e81660b44b6a9b1ae4f98cfaeab1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:31.323808 8776 leader_election.cc:302] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:31.324023 14636 raft_consensus.cc:2781] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.324190 14636 raft_consensus.cc:686] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:31.324330 14636 consensus_queue.cc:237] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:31.326587 6150 catalog_manager.cc:5484] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.362993 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.374382 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:31.378939 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.379304 14656 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:31.379526 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.379807 14658 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.379845 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:31.380092 14657 tablet.cc:1620] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.384510 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 2)
W20240503 07:59:31.423069 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.486069 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:31.490602 14656 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.491501 14657 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.491513 14658 tablet.cc:1722] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:31.497238 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0] alter complete (version 3)
I20240503 07:59:31.501936 14523 raft_consensus.cc:482] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.502019 14523 raft_consensus.cc:504] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.502400 14523 leader_election.cc:288] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:31.502647 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.502869 6298 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.503155 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:31.503283 8975 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:31.503566 6627 leader_election.cc:302] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
W20240503 07:59:31.503705 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.503777 14523 raft_consensus.cc:2781] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.503818 14523 raft_consensus.cc:482] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.503860 14523 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.504690 14523 raft_consensus.cc:504] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.505020 14523 leader_election.cc:288] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:31.505273 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:31.505291 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702e72e3876a453a83ea779d4f5239d7" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.505367 8975 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.505411 6298 raft_consensus.cc:3037] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.506021 8975 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.506116 6298 raft_consensus.cc:2454] T 702e72e3876a453a83ea779d4f5239d7 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:31.506242 6627 leader_election.cc:302] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:31.506395 14523 raft_consensus.cc:2781] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.506465 14523 raft_consensus.cc:686] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:31.506579 14523 consensus_queue.cc:237] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:31.508354 6150 catalog_manager.cc:5484] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:31.517640 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:31.549204 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:31.558929 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:31.579638 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.600298 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.600503 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:31.602821 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.603338 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:31.603452 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:31.603916 14665 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:31.604048 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5511ac79f64b7ab3edbdafe8df3ad9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:31 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:31.604099 14664 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:31.604112 14665 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:31.604228 14664 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.604373 14664 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.604391 14665 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.604915 14665 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.604984 14664 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.606931 14665 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.606941 14664 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.607002 14665 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.607003 14664 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.607252 14664 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:31.607333 14665 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:31.607820 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
I20240503 07:59:31.608050 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
I20240503 07:59:31.608930 14666 tablet_replica.cc:330] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:31.609126 14666 raft_consensus.cc:2229] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:31.609242 14666 raft_consensus.cc:2258] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:31.609881 14666 ts_tablet_manager.cc:1875] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:31.612370 14666 ts_tablet_manager.cc:1888] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:31.612488 14666 log.cc:1196] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fb5511ac79f64b7ab3edbdafe8df3ad9
I20240503 07:59:31.612807 14666 ts_tablet_manager.cc:1909] T fb5511ac79f64b7ab3edbdafe8df3ad9 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:31.613574 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet fb5511ac79f64b7ab3edbdafe8df3ad9 (table rebalancer_dynamic_table_new_name [id=ebf2543a943f483b951c6c2c656ba8b0]) successfully deleted
W20240503 07:59:31.654539 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.674054 14604 consensus_queue.cc:1035] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.676633 14661 consensus_queue.cc:1035] T 054c5cb68f4b47809b2c0dcfc3281d95 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.704663 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:31.709286 6279 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.709318 6540 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.709528 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.709528 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.711524 14641 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:31.711834 8560 tablet_service.cc:1467] Processing CreateTablet for tablet fe07f18e6897468596e83277e4cc7459 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]), partition=
I20240503 07:59:31.712049 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe07f18e6897468596e83277e4cc7459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:31.712389 14641 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.712929 14626 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:31.713392 14641 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:31.713470 14641 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:31.713725 14626 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.713894 14641 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.713989 14641 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.714020 14641 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.714102 14674 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:31.714133 14641 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.714383 14641 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.715058 14626 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:31.715138 14626 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:31.715415 14674 tablet_bootstrap.cc:654] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:31.715507 14626 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.715577 14626 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.715595 14626 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.715687 14626 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.715939 14626 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:31.717973 14674 tablet_bootstrap.cc:492] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:31.718080 14674 ts_tablet_manager.cc:1367] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:31.718509 14674 raft_consensus.cc:348] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.718600 14674 raft_consensus.cc:374] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:31.718621 14674 raft_consensus.cc:729] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:31.718711 14674 consensus_queue.cc:260] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.718945 14674 ts_tablet_manager.cc:1398] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:31.733991 14636 consensus_queue.cc:1035] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.736522 14291 consensus_queue.cc:1035] T bc77e81660b44b6a9b1ae4f98cfaeab1 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.788005 14634 raft_consensus.cc:482] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:31.788107 14634 raft_consensus.cc:504] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.788622 14634 leader_election.cc:288] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
W20240503 07:59:31.789314 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:31.789363 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:31.789417 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:31.789482 6298 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:31.789515 6560 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:31.789737 8514 leader_election.cc:302] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.789935 14634 raft_consensus.cc:2781] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:31.789999 14634 raft_consensus.cc:482] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:31.790036 14634 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.790841 14634 raft_consensus.cc:504] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.791144 14634 leader_election.cc:288] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:31.791438 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:31.791414 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe07f18e6897468596e83277e4cc7459" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:31.791541 6560 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.791550 6298 raft_consensus.cc:3037] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:31.792384 6560 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:31.792394 6298 raft_consensus.cc:2454] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:31.792656 8514 leader_election.cc:302] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:31.792830 14634 raft_consensus.cc:2781] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:31.792989 14634 raft_consensus.cc:686] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:31.793130 14634 consensus_queue.cc:237] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:31.795190 6156 catalog_manager.cc:5484] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:31.805665 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:31.838999 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:31.883277 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:31.911718 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:31.915369 6560 raft_consensus.cc:1261] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.915475 6298 raft_consensus.cc:1261] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:31.915838 14680 consensus_queue.cc:1035] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.916016 14634 consensus_queue.cc:1035] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.918876 6156 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.918888 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.919169 14658 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:31.919273 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.919544 14657 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:31.920169 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.920822 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.923856 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 1)
I20240503 07:59:31.925127 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:31.925503 14685 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:31.926614 6154 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8]). Expected version 1 got 0
I20240503 07:59:31.969439 14523 consensus_queue.cc:1035] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:31.972348 14564 consensus_queue.cc:1035] T 702e72e3876a453a83ea779d4f5239d7 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.021101 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:32.026079 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.027531 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027663 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027820 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.027827 6408 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.027897 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.027827 6670 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.028069 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.028069 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.030298 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.030392 14674 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.030512 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.030601 14690 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:32.031075 14691 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.031302 14674 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.031911 14691 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.032817 14689 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.033443 14690 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.033516 6540 tablet_service.cc:1467] Processing CreateTablet for tablet ad7c5d1c8b9f4674bd1adee106613938 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.033787 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad7c5d1c8b9f4674bd1adee106613938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.033804 14689 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.034794 14690 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:32.034880 14690 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.035214 14690 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.035302 14690 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.035579 14626 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.035332 14690 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.035755 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.035790 14690 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.035960 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.036167 14690 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.036453 14626 tablet_bootstrap.cc:654] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.037189 14691 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.037258 14691 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:32.037552 14626 tablet_bootstrap.cc:492] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.037617 14626 ts_tablet_manager.cc:1367] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:32.037593 14691 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.037667 14691 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.037686 14691 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.037786 14691 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.037998 14626 raft_consensus.cc:348] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.038090 14626 raft_consensus.cc:374] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.038120 14626 raft_consensus.cc:729] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.038125 14691 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.038251 14691 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.038221 14626 consensus_queue.cc:260] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.038478 14626 ts_tablet_manager.cc:1398] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.038637 14626 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.039077 14691 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.040225 14626 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.040812 14691 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.040905 14691 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.041287 14691 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.041384 14691 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.041416 14691 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.041517 14691 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.041790 14691 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.042078 14626 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.042145 14626 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:32.042568 14626 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.042659 14626 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.042687 14626 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.042783 14626 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
W20240503 07:59:32.042833 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:32.043026 14626 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.045980 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.046249 14658 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:32.047008 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 770eef868fe84945bf6b0a0e73affd13 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.047273 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770eef868fe84945bf6b0a0e73affd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.047008 8822 tablet_service.cc:1467] Processing CreateTablet for tablet d457837d7f68418699ac7b1f7cdc9100 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=c7220cde22544802ad5cd9c10137f0ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.049048 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.049228 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d457837d7f68418699ac7b1f7cdc9100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.049319 14685 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.049407 14699 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.050483 14699 tablet_bootstrap.cc:654] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.053159 14699 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.053297 14699 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.054178 14699 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.054265 14699 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.054291 14699 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.054437 14699 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.054798 14699 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.055011 14699 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.056012 14699 tablet_bootstrap.cc:654] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.056869 14699 tablet_bootstrap.cc:492] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.056978 14699 ts_tablet_manager.cc:1367] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.057307 14699 raft_consensus.cc:348] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.057391 14699 raft_consensus.cc:374] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.057420 14699 raft_consensus.cc:729] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.057547 14699 consensus_queue.cc:260] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.057816 14699 ts_tablet_manager.cc:1398] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:32.059487 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 2)
I20240503 07:59:32.061543 14674 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.061628 14674 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.000s
I20240503 07:59:32.061658 14689 tablet_bootstrap.cc:492] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.061750 14689 ts_tablet_manager.cc:1367] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 07:59:32.062009 14674 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062085 14674 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.062104 14674 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.062192 14674 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062209 14689 raft_consensus.cc:348] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062307 14689 raft_consensus.cc:374] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.062337 14689 raft_consensus.cc:729] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.062438 14674 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.062463 14689 consensus_queue.cc:260] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.062727 14689 ts_tablet_manager.cc:1398] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.063661 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.063939 14657 tablet.cc:1620] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
W20240503 07:59:32.130388 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:32.133592 14564 raft_consensus.cc:482] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.133697 14564 raft_consensus.cc:504] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.134109 14564 leader_election.cc:288] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.134402 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.134512 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:32.134534 6560 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:32.134631 6430 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:32.135115 6625 leader_election.cc:302] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.135481 14564 raft_consensus.cc:2781] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.135543 14564 raft_consensus.cc:482] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.135582 14564 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.136545 14564 raft_consensus.cc:504] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.136904 14564 leader_election.cc:288] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.137315 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.137416 6560 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.138127 6560 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:32.138427 6625 leader_election.cc:302] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.138814 14564 raft_consensus.cc:2781] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.138902 14564 raft_consensus.cc:686] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:32.139024 14564 consensus_queue.cc:237] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.139601 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad7c5d1c8b9f4674bd1adee106613938" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:32.139707 6430 raft_consensus.cc:3037] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.140367 6430 raft_consensus.cc:2454] T ad7c5d1c8b9f4674bd1adee106613938 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:32.141634 6154 catalog_manager.cc:5484] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.150163 14636 raft_consensus.cc:482] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.150254 14636 raft_consensus.cc:504] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.150655 14636 leader_election.cc:288] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.151057 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.151170 6560 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:32.151592 8776 leader_election.cc:302] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.151840 14636 raft_consensus.cc:2781] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.151903 14636 raft_consensus.cc:482] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.151953 14636 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.152871 14636 raft_consensus.cc:504] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.153460 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:32.153548 6429 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.153587 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.153673 6560 raft_consensus.cc:3037] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.154486 6560 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:32.154505 6429 raft_consensus.cc:2454] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:32.154735 8776 leader_election.cc:302] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:32.154882 14291 raft_consensus.cc:2781] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.153381 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d457837d7f68418699ac7b1f7cdc9100" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:32.154946 14291 raft_consensus.cc:686] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:32.154983 6430 raft_consensus.cc:2362] T d457837d7f68418699ac7b1f7cdc9100 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1. Re-sending same reply.
I20240503 07:59:32.155045 14291 consensus_queue.cc:237] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:32.156158 14636 leader_election.cc:288] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:32.157369 6156 catalog_manager.cc:5484] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.161342 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:32.167025 14658 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.168272 14657 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.169487 14685 tablet.cc:1722] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 07:59:32.173662 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.174958 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8] alter complete (version 3)
I20240503 07:59:32.191150 14680 raft_consensus.cc:482] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.191252 14680 raft_consensus.cc:504] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.191669 14680 leader_election.cc:288] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.191927 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.192005 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:32.192118 8842 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.192127 8975 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.192409 8516 leader_election.cc:302] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.192636 14680 raft_consensus.cc:2781] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.192682 14680 raft_consensus.cc:482] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.192718 14680 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.193434 14680 raft_consensus.cc:504] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.193720 14680 leader_election.cc:288] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.194027 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:32.194029 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770eef868fe84945bf6b0a0e73affd13" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.194125 8975 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.194126 8842 raft_consensus.cc:3037] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.194965 8842 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.194965 8975 raft_consensus.cc:2454] T 770eef868fe84945bf6b0a0e73affd13 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.195225 8516 leader_election.cc:302] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.195524 14680 raft_consensus.cc:2781] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.195605 14680 raft_consensus.cc:686] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:32.195724 14680 consensus_queue.cc:237] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:32.197853 6154 catalog_manager.cc:5484] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.278659 6154 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.278791 6154 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.280748 6154 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.281301 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:32.281437 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:32.281436 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet fe07f18e6897468596e83277e4cc7459 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:32.281594 14706 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:32.281739 14708 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:32.281793 14706 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.281919 14707 tablet_replica.cc:330] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:32.281955 14706 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282055 14707 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.282063 14708 raft_consensus.cc:2229] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:32.282166 14707 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282271 14708 raft_consensus.cc:2258] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.282681 14707 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.282660 14706 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.282817 14708 ts_tablet_manager.cc:1875] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.285202 14708 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.285224 14706 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.285288 14706 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.285305 14708 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.285686 14708 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:32.285781 14706 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:32.286343 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
I20240503 07:59:32.286509 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
I20240503 07:59:32.287096 14707 ts_tablet_manager.cc:1888] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.287170 14707 log.cc:1196] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe07f18e6897468596e83277e4cc7459
I20240503 07:59:32.287520 14707 ts_tablet_manager.cc:1909] T fe07f18e6897468596e83277e4cc7459 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:32.288209 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet fe07f18e6897468596e83277e4cc7459 (table rebalancer_dynamic_table_new_name [id=bff26081edbb4d1d916ee3cbc35dd3f8]) successfully deleted
W20240503 07:59:32.341648 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.382753 6154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:32.388303 8955 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388396 6670 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388574 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.388648 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.388689 6279 tablet_service.cc:1467] Processing CreateTablet for tablet ef06769a02954deb8f2c219f485fdea1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]), partition=
I20240503 07:59:32.388921 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef06769a02954deb8f2c219f485fdea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.390411 14689 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.390852 14690 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:32.391393 14689 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.391723 14690 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.392071 14710 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.392791 14689 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.392890 14689 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.392918 14690 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:32.392982 14690 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:32.393321 14689 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393361 14690 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393419 14689 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.393448 14690 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.393448 14689 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.393476 14690 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.393573 14690 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393563 14689 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.393822 14690 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.393834 14689 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.393896 14710 tablet_bootstrap.cc:654] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.396108 14710 tablet_bootstrap.cc:492] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.396203 14710 ts_tablet_manager.cc:1367] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.396711 14710 raft_consensus.cc:348] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.396840 14710 raft_consensus.cc:374] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.396920 14710 raft_consensus.cc:729] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.397048 14710 consensus_queue.cc:260] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.398068 14710 ts_tablet_manager.cc:1398] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:32.403946 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:32.421985 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.438863 14573 raft_consensus.cc:482] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.438963 14573 raft_consensus.cc:504] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.439309 14573 leader_election.cc:288] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:32.439553 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.439697 8975 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.439829 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:32.439942 6690 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.439965 6234 leader_election.cc:302] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:32.440121 14573 raft_consensus.cc:2781] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.440188 14573 raft_consensus.cc:482] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.440233 14573 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.441376 14573 raft_consensus.cc:504] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:32.441748 14573 leader_election.cc:288] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:32.444406 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.444519 8975 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.444625 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef06769a02954deb8f2c219f485fdea1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:32.444741 6690 raft_consensus.cc:3037] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.445641 6690 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.446012 6234 leader_election.cc:302] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:32.446193 14573 raft_consensus.cc:2781] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.446261 14573 raft_consensus.cc:686] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:32.446264 8975 raft_consensus.cc:2454] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.446393 14573 consensus_queue.cc:237] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
W20240503 07:59:32.447997 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.448885 6156 catalog_manager.cc:5484] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:32.491298 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.492318 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.541538 14291 consensus_queue.cc:1035] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.547068 14291 consensus_queue.cc:1035] T d457837d7f68418699ac7b1f7cdc9100 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.555138 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:32.558930 6691 raft_consensus.cc:1261] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.558954 8976 raft_consensus.cc:1261] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c167a0a433d34b74affed13cbf96e302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:32.559289 14709 consensus_queue.cc:1035] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.559448 14573 consensus_queue.cc:1035] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.562260 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.562546 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.563038 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.563362 14685 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.563668 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.564891 14725 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:32.565802 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.566303 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41]). Expected version 1 got 0
I20240503 07:59:32.567458 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:32.567744 14724 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:32.569547 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41] alter complete (version 1)
I20240503 07:59:32.590694 14680 consensus_queue.cc:1035] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.593869 14727 consensus_queue.cc:1035] T 770eef868fe84945bf6b0a0e73affd13 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:32.636518 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:32.643007 14523 consensus_queue.cc:1035] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:32.645790 14427 consensus_queue.cc:1035] T ad7c5d1c8b9f4674bd1adee106613938 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:32.647678 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.652133 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.670861 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:32.675283 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.675585 14685 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:32.676337 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.676853 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:32.677088 14725 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:32.677124 14724 tablet.cc:1620] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:32.682986 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=333d433f0105408499c233f4e2587a41] alter complete (version 2)
W20240503 07:59:32.717115 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.725323 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:32.730978 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731298 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.731336 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731585 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.731632 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 9a7fc8787257459ca681e9dd03fed2c4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.731853 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a7fc8787257459ca681e9dd03fed2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.733791 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.733872 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.734002 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.734133 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.734156 14732 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:32.734325 8559 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.734539 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.735273 14732 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.735976 6540 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.736184 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.736388 14734 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.736891 14710 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.736900 14731 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.737504 14734 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.737942 14731 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.738770 14734 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.738811 14710 tablet_bootstrap.cc:654] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.738864 14734 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.739220 14731 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.739308 14731 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.739964 14710 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.739918 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.740042 14710 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.740001 8955 tablet_service.cc:1467] Processing CreateTablet for tablet b138bad3526b4e37b0f08e3efc3e3766 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=76418df416254ab3b8de63d9dfbf8a41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:32.740170 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1ab1f59dbc41d482e7a51f7b72ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.740247 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b138bad3526b4e37b0f08e3efc3e3766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:32.740428 14710 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.740533 14710 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.740562 14710 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.740664 14710 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.741205 14710 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.742345 14689 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:32.739284 14734 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.742553 14734 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.742588 14734 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.742702 14734 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.742873 14732 tablet_bootstrap.cc:492] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:32.742949 14732 ts_tablet_manager.cc:1367] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:32.742949 14734 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240503 07:59:32.743093 14734 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:32.743201 14689 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.743558 14732 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.743661 14732 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.743687 14732 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.743790 14732 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.743978 14734 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.744051 14732 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:32.745770 14734 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:32.745921 14734 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:32.746390 14734 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.746516 14734 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.746577 14734 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.746706 14689 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:32.746734 14734 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.746790 14689 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:32.747118 14734 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.747193 14689 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.747289 14689 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.747282 14710 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:32.747315 14689 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.747422 14689 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.747668 14689 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.748214 14710 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:32.748682 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.749053 14731 raft_consensus.cc:348] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.749207 14710 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:32.749284 14710 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:32.749655 14710 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.749755 14710 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.749784 14710 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.749929 14710 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.750207 14710 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.750995 14731 raft_consensus.cc:374] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.751065 14731 raft_consensus.cc:729] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.751217 14731 consensus_queue.cc:260] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.752270 14731 ts_tablet_manager.cc:1398] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 07:59:32.752470 14731 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:32.753329 14731 tablet_bootstrap.cc:654] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.754104 14738 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:32.754936 14738 tablet_bootstrap.cc:654] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:32.755918 14731 tablet_bootstrap.cc:492] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:32.756040 14731 ts_tablet_manager.cc:1367] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:32.756490 14731 raft_consensus.cc:348] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.756580 14731 raft_consensus.cc:374] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.756609 14731 raft_consensus.cc:729] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.756727 14731 consensus_queue.cc:260] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.756981 14731 ts_tablet_manager.cc:1398] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.757437 14738 tablet_bootstrap.cc:492] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:32.757508 14738 ts_tablet_manager.cc:1367] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:32.758024 14738 raft_consensus.cc:348] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.758126 14738 raft_consensus.cc:374] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:32.758154 14738 raft_consensus.cc:729] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:32.758291 14738 consensus_queue.cc:260] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.758637 14738 ts_tablet_manager.cc:1398] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:32.771960 14727 raft_consensus.cc:482] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.772032 14727 raft_consensus.cc:504] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.772296 14727 leader_election.cc:288] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:32.772629 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:32.772629 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:32.772739 6559 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.772739 8976 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:32.773090 8516 leader_election.cc:302] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.773253 14727 raft_consensus.cc:2781] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.773296 14727 raft_consensus.cc:482] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.773331 14727 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.774111 14727 raft_consensus.cc:504] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.774398 14727 leader_election.cc:288] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:32.774647 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:32.774662 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b138bad3526b4e37b0f08e3efc3e3766" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:32.774744 6559 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.774756 8976 raft_consensus.cc:3037] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.775506 6559 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.775668 8976 raft_consensus.cc:2454] T b138bad3526b4e37b0f08e3efc3e3766 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:32.775759 8514 leader_election.cc:302] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.775938 14727 raft_consensus.cc:2781] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.776010 14727 raft_consensus.cc:686] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:32.776134 14727 consensus_queue.cc:237] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:32.778544 6150 catalog_manager.cc:5484] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:32.786098 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:32.790575 14685 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.791496 14725 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.791808 14724 tablet.cc:1722] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:32.795285 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41] alter complete (version 3)
W20240503 07:59:32.836045 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.851310 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:32.864740 14723 raft_consensus.cc:482] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:32.864866 14723 raft_consensus.cc:504] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.865343 14723 leader_election.cc:288] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.865571 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:32.865628 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:32.865700 8581 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.865756 8843 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:32.865967 6232 leader_election.cc:302] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.866137 14723 raft_consensus.cc:2781] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:32.866196 14723 raft_consensus.cc:482] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:32.866240 14723 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.867004 14723 raft_consensus.cc:504] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.867292 14723 leader_election.cc:288] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:32.867568 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:32.867583 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1ab1f59dbc41d482e7a51f7b72ffd4" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:32.867671 8843 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.867669 8581 raft_consensus.cc:3037] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:32.868392 8843 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.868428 8581 raft_consensus.cc:2454] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:32.868651 6232 leader_election.cc:302] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:32.869009 14723 raft_consensus.cc:2781] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:32.869302 14723 raft_consensus.cc:686] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:32.869458 14723 consensus_queue.cc:237] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:32.871479 6156 catalog_manager.cc:5484] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:32.897997 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.898140 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:32.900192 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.900759 6278 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:32.900832 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:32.900945 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ef06769a02954deb8f2c219f485fdea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:32 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:32.901304 14746 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:32.901319 14748 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:32.901458 14748 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.901490 14746 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:32.901585 14748 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.901672 14746 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.902112 14747 tablet_replica.cc:330] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:32.902200 14748 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.902246 14747 raft_consensus.cc:2229] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:32.902200 14746 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.902398 14747 raft_consensus.cc:2258] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:32.902890 14747 ts_tablet_manager.cc:1875] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:32.904539 14748 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.904551 14746 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.904611 14746 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.904640 14748 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.904918 14748 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:32.904969 14746 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:32.905370 14747 ts_tablet_manager.cc:1888] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:32.905479 14747 log.cc:1196] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef06769a02954deb8f2c219f485fdea1
I20240503 07:59:32.905623 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
I20240503 07:59:32.905781 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
I20240503 07:59:32.906715 14747 ts_tablet_manager.cc:1909] T ef06769a02954deb8f2c219f485fdea1 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:32.907570 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet ef06769a02954deb8f2c219f485fdea1 (table rebalancer_dynamic_table_new_name [id=333d433f0105408499c233f4e2587a41]) successfully deleted
W20240503 07:59:32.960942 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.963039 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:32.968410 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:32.995208 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 07:59:33.002578 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:33.008710 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.008895 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.008991 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.009078 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.011073 14734 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.011234 14356 raft_consensus.cc:482] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.011291 14750 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.011309 14356 raft_consensus.cc:504] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.011750 14356 leader_election.cc:288] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:33.011919 14734 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.012017 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 659a99860b3e44daad55dffa53296b44 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]), partition=
I20240503 07:59:33.012207 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 659a99860b3e44daad55dffa53296b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.012662 14750 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.013778 14734 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.013871 14734 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.014290 14734 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.014381 14734 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.014413 14734 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.014472 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:33.014523 14734 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.014586 6298 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:33.014798 14734 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.014847 6494 leader_election.cc:302] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.014987 14356 raft_consensus.cc:2781] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.015028 14356 raft_consensus.cc:482] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.015061 14356 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.015079 14731 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:33.015867 14731 tablet_bootstrap.cc:654] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.015910 14356 raft_consensus.cc:504] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.016199 14356 leader_election.cc:288] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:33.016525 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:33.016619 6298 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.017535 6298 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:33.017779 6494 leader_election.cc:302] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.018090 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:33.018200 6429 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:33.019235 6430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a7fc8787257459ca681e9dd03fed2c4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:33.019326 6430 raft_consensus.cc:3037] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.020154 6430 raft_consensus.cc:2454] T 9a7fc8787257459ca681e9dd03fed2c4 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:33.020345 14731 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:33.020429 14731 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:33.022009 14731 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.022102 14731 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.022130 14731 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.022240 14731 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.022531 14731 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.022696 14749 raft_consensus.cc:2781] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.022836 14749 raft_consensus.cc:686] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:33.022953 14749 consensus_queue.cc:237] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.026618 14750 tablet_bootstrap.cc:492] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.026700 14750 ts_tablet_manager.cc:1367] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:33.026594 6156 catalog_manager.cc:5484] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:33.027122 14750 raft_consensus.cc:348] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.027216 14750 raft_consensus.cc:374] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.027244 14750 raft_consensus.cc:729] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.027364 14750 consensus_queue.cc:260] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.027623 14750 ts_tablet_manager.cc:1398] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:33.064486 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:33.152536 14759 consensus_queue.cc:1035] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.159993 14680 consensus_queue.cc:1035] T b138bad3526b4e37b0f08e3efc3e3766 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.184214 14760 raft_consensus.cc:482] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.184330 14760 raft_consensus.cc:504] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.184710 14760 leader_election.cc:288] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:33.184985 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:33.185055 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:33.185119 8581 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.185197 6559 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.185405 6625 leader_election.cc:302] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.185552 14760 raft_consensus.cc:2781] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.185612 14760 raft_consensus.cc:482] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.185721 14760 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.187115 14760 raft_consensus.cc:504] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.187393 14760 leader_election.cc:288] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:33.187638 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:33.187731 8581 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.187707 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "659a99860b3e44daad55dffa53296b44" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:33.187822 6559 raft_consensus.cc:3037] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.188450 8581 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.188508 6559 raft_consensus.cc:2454] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.188688 6625 leader_election.cc:302] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.188828 14760 raft_consensus.cc:2781] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.188915 14760 raft_consensus.cc:686] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:33.189064 14760 consensus_queue.cc:237] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:33.191566 6154 catalog_manager.cc:5484] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:33.192631 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 07:59:33.300232 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.301235 14723 consensus_queue.cc:1035] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.304814 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:33.305006 14723 consensus_queue.cc:1035] T 5e1ab1f59dbc41d482e7a51f7b72ffd4 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.309516 6560 raft_consensus.cc:1261] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.309609 8581 raft_consensus.cc:1261] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:33.309939 14760 consensus_queue.cc:1035] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.310201 14760 consensus_queue.cc:1035] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.312400 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.312819 14773 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:33.314157 6150 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.315445 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.315861 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.316139 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.316550 14775 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:33.318086 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 1)
I20240503 07:59:33.318779 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db]). Expected version 1 got 0
I20240503 07:59:33.331682 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:33.331971 14774 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.419572 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:33.424103 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.424448 14773 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:33.424979 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.425254 14775 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:33.426627 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:33.427524 14774 tablet.cc:1620] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:33.429687 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 2)
W20240503 07:59:33.478113 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:33.533394 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:33.538445 14774 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.538642 14773 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.541018 14775 tablet.cc:1722] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:33.544425 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db] alter complete (version 3)
I20240503 07:59:33.562163 14356 consensus_queue.cc:1035] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.565003 14756 consensus_queue.cc:1035] T 9a7fc8787257459ca681e9dd03fed2c4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:33.596261 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:33.601528 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.601787 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.602320 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602581 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.602721 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602952 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.603914 14781 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.604058 6277 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.602320 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.604252 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.604399 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.604636 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce814e0404442bbe40a36dfa3e5e30 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.604856 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ce814e0404442bbe40a36dfa3e5e30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.605139 14781 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.606770 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6986da39c86f45fab9d05ce97fdc1b73 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.606988 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6986da39c86f45fab9d05ce97fdc1b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.607242 14783 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:33.607182 8955 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.607431 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.608052 14783 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.608942 14786 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.610517 14782 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.610831 14786 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.612586 6670 tablet_service.cc:1467] Processing CreateTablet for tablet d90468911a2846b199e01ec1a399fb7a (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=8b9b081f91cf4e8a8cfe8dd5f0302bd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:33.612860 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90468911a2846b199e01ec1a399fb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.613337 14782 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.615271 14787 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:33.616855 14787 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.618016 14787 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:33.618104 14787 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.618549 14787 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.618654 14787 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.618736 14787 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.618871 14787 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.619202 14787 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.620363 14782 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.620507 14782 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:33.620931 14786 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.621039 14786 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 07:59:33.621043 14782 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621136 14782 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.621170 14782 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.621282 14782 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621515 14786 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.621610 14786 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.621642 14786 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.621747 14786 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.622047 14786 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.621564 14782 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.624943 14782 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.625967 14783 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:33.626039 14783 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 07:59:33.626202 14782 tablet_bootstrap.cc:654] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.626435 14783 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.626543 14783 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.626582 14783 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.626690 14783 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.626956 14783 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.627025 14781 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.627094 14781 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.000s
I20240503 07:59:33.627117 14783 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:33.627525 14781 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.627616 14781 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.627645 14781 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.627911 14783 tablet_bootstrap.cc:654] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.631176 14781 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.631508 14781 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:33.631685 14781 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.633728 14781 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.636376 14781 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.637800 14781 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:33.638121 14782 tablet_bootstrap.cc:492] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.638235 14782 ts_tablet_manager.cc:1367] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:33.638484 14781 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.638628 14781 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.638821 14781 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.638957 14781 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.639084 14782 raft_consensus.cc:348] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.639209 14782 raft_consensus.cc:374] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.639267 14782 raft_consensus.cc:729] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.639434 14781 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.639428 14782 consensus_queue.cc:260] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.639710 14782 ts_tablet_manager.cc:1398] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.639880 14782 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:33.640705 14782 tablet_bootstrap.cc:654] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.641466 14783 tablet_bootstrap.cc:492] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:33.641541 14783 ts_tablet_manager.cc:1367] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:33.641899 14783 raft_consensus.cc:348] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.641978 14783 raft_consensus.cc:374] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.642002 14783 raft_consensus.cc:729] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.642109 14783 consensus_queue.cc:260] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.642323 14783 ts_tablet_manager.cc:1398] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.645115 14782 tablet_bootstrap.cc:492] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:33.645223 14782 ts_tablet_manager.cc:1367] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:33.645776 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.645814 14782 raft_consensus.cc:348] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.645943 14782 raft_consensus.cc:374] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.646003 14782 raft_consensus.cc:729] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.645978 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:33.646139 14782 consensus_queue.cc:260] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.646528 14782 ts_tablet_manager.cc:1398] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.648501 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.649104 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:33.649251 6670 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:33.649318 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 659a99860b3e44daad55dffa53296b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:33 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:33.649397 14795 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:33.649526 14795 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.649610 14797 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:33.649653 14795 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.649827 14797 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:33.650002 14797 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.650195 14795 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.650301 14796 tablet_replica.cc:330] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:33.650432 14796 raft_consensus.cc:2229] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:33.650609 14796 raft_consensus.cc:2258] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:33.650611 14797 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.651042 14796 ts_tablet_manager.cc:1875] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:33.652565 14795 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.652680 14795 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653074 14795 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:33.653394 14796 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.653604 14797 ts_tablet_manager.cc:1888] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:33.653631 14796 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653748 14797 log.cc:1196] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/659a99860b3e44daad55dffa53296b44
I20240503 07:59:33.653786 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
I20240503 07:59:33.653914 14796 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:33.654175 14797 ts_tablet_manager.cc:1909] T 659a99860b3e44daad55dffa53296b44 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:33.654520 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
I20240503 07:59:33.654877 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 659a99860b3e44daad55dffa53296b44 (table rebalancer_dynamic_table_new_name [id=45a4c877fd5d4bef8d40a79de11e35db]) successfully deleted
W20240503 07:59:33.671602 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.713645 14723 raft_consensus.cc:482] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.713747 14723 raft_consensus.cc:504] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.714072 14723 leader_election.cc:288] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.714397 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:33.714421 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:33.714531 8843 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.714532 8581 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.714779 6232 leader_election.cc:302] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.714964 14723 raft_consensus.cc:2781] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.715030 14723 raft_consensus.cc:482] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.715056 14723 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.716013 14723 raft_consensus.cc:504] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.716327 14723 leader_election.cc:288] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.716629 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:33.716748 8843 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.716725 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6986da39c86f45fab9d05ce97fdc1b73" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:33.716852 8581 raft_consensus.cc:3037] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.717715 8581 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.717710 8843 raft_consensus.cc:2454] T 6986da39c86f45fab9d05ce97fdc1b73 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.717988 6234 leader_election.cc:302] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:33.718199 14723 raft_consensus.cc:2781] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.718282 14723 raft_consensus.cc:686] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:33.718429 14723 consensus_queue.cc:237] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.720522 6155 catalog_manager.cc:5484] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:33.749773 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:33.754778 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.754784 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.755031 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.755121 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.756755 14800 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:33.756909 14786 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:33.757866 14786 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.757891 14800 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.758692 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 4030f6767afe4113b35a1ad1fd668912 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]), partition=
I20240503 07:59:33.758870 14786 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:33.758905 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4030f6767afe4113b35a1ad1fd668912. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:33.758960 14786 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:33.759075 14800 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:33.759157 14800 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:33.759423 14786 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759526 14786 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.759565 14786 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.759554 14800 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759665 14800 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.759698 14800 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.759678 14786 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.759935 14786 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:33.759802 14800 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.760607 14781 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:33.760746 14800 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:33.762642 14781 tablet_bootstrap.cc:654] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:33.763707 14781 tablet_bootstrap.cc:492] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:33.763803 14781 ts_tablet_manager.cc:1367] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:33.764201 14781 raft_consensus.cc:348] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.764281 14781 raft_consensus.cc:374] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:33.764307 14781 raft_consensus.cc:729] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:33.764412 14781 consensus_queue.cc:260] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:33.764675 14781 ts_tablet_manager.cc:1398] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:33.772929 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:33.791182 14776 raft_consensus.cc:482] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.791281 14776 raft_consensus.cc:504] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.791707 14776 leader_election.cc:288] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:33.792004 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:33.792032 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:33.792130 8976 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.792145 6296 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:33.792371 6625 leader_election.cc:302] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.792541 14776 raft_consensus.cc:2781] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.792615 14776 raft_consensus.cc:482] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.792665 14776 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.793779 14776 raft_consensus.cc:504] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.794169 14776 leader_election.cc:288] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:33.794430 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:33.794430 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90468911a2846b199e01ec1a399fb7a" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:33.794540 8976 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.794540 6296 raft_consensus.cc:3037] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.795428 8976 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.795428 6296 raft_consensus.cc:2454] T d90468911a2846b199e01ec1a399fb7a P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:33.795676 6625 leader_election.cc:302] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.795856 14776 raft_consensus.cc:2781] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.795928 14776 raft_consensus.cc:686] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:33.796066 14776 consensus_queue.cc:237] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:33.798311 6155 catalog_manager.cc:5484] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:33.830708 14723 raft_consensus.cc:482] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:33.830801 14723 raft_consensus.cc:504] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.831140 14723 leader_election.cc:288] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.831455 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:33.831555 8843 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.831740 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:33.831777 6234 leader_election.cc:302] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.831857 6690 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:33.832032 14723 raft_consensus.cc:2781] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:33.832094 14723 raft_consensus.cc:482] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:33.832134 14723 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.832845 14723 raft_consensus.cc:504] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.833110 14723 leader_election.cc:288] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:33.833370 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:33.833462 6690 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.834103 6690 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.834318 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ce814e0404442bbe40a36dfa3e5e30" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:33.834340 6234 leader_election.cc:302] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:33.834415 8843 raft_consensus.cc:3037] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:33.835047 8843 raft_consensus.cc:2454] T 55ce814e0404442bbe40a36dfa3e5e30 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:33.835253 14723 raft_consensus.cc:2781] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:33.835337 14723 raft_consensus.cc:686] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:33.835445 14723 consensus_queue.cc:237] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:33.837549 6155 catalog_manager.cc:5484] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:33.977959 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:33.989336 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.011155 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:34.040060 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.061609 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.075883 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.136077 14759 raft_consensus.cc:482] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.136190 14759 raft_consensus.cc:504] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.136696 14759 leader_election.cc:288] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:34.136981 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:34.136987 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:34.137096 6690 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:34.137121 6559 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:34.137341 8516 leader_election.cc:302] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:34.137552 14759 raft_consensus.cc:2781] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.137614 14759 raft_consensus.cc:482] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.137667 14759 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.138671 14759 raft_consensus.cc:504] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.138967 14759 leader_election.cc:288] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:34.139346 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:34.139387 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4030f6767afe4113b35a1ad1fd668912" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:34.139442 6690 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.139511 6559 raft_consensus.cc:3037] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.140200 6690 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:34.140470 8516 leader_election.cc:302] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:34.140514 6559 raft_consensus.cc:2454] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:34.140628 14759 raft_consensus.cc:2781] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.140707 14759 raft_consensus.cc:686] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:34.140866 14759 consensus_queue.cc:237] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:34.143806 6155 catalog_manager.cc:5484] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:34.177676 14427 consensus_queue.cc:1035] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.180289 14776 consensus_queue.cc:1035] T d90468911a2846b199e01ec1a399fb7a P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.194424 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:34.208214 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.260418 14806 consensus_queue.cc:1035] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.262837 14723 consensus_queue.cc:1035] T 6986da39c86f45fab9d05ce97fdc1b73 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.305518 6154 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:34.309371 6690 raft_consensus.cc:1261] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.309455 6559 raft_consensus.cc:1261] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:34.309861 14759 consensus_queue.cc:1035] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.310128 14759 consensus_queue.cc:1035] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.313726 6154 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.315527 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.317348 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.317577 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.317678 14821 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:34.317739 6154 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.317982 14822 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:34.320755 6154 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.321872 6155 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00]). Expected version 1 got 0
I20240503 07:59:34.325567 6154 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 1)
I20240503 07:59:34.329152 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:34.329602 14823 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.353701 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:34.358903 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359182 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.359223 6410 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359475 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.359596 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.359825 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.361136 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.361204 14824 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:34.361362 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.361783 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 757545bdc0e3426fa36c7ffcbd4189e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.361990 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 757545bdc0e3426fa36c7ffcbd4189e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.362376 14824 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.363554 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.363662 14824 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:34.363729 14824 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.363790 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.364199 14824 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.364365 14824 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.364400 14824 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.364754 14824 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.365124 14824 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.365324 14828 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:34.365573 14825 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.366524 14828 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.366649 14825 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.366616 6670 tablet_service.cc:1467] Processing CreateTablet for tablet 96720ba75f544ee18872449e7b26967f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.366864 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96720ba75f544ee18872449e7b26967f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.367615 14828 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:34.367688 14828 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:34.368106 14828 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.368193 14828 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.368217 14828 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.368306 14828 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.368558 14828 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.368957 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.369129 14832 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:34.369177 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.369975 14832 tablet_bootstrap.cc:654] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.370980 14832 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:34.371052 14832 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.371480 14832 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.371587 14832 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.371620 14832 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.371734 14832 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.372016 14832 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.372001 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6246801a888a4ff3b280b812de0318e1 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=a509464cccc1496bad04e843cc16c52b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:34.372162 14832 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:34.372233 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6246801a888a4ff3b280b812de0318e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.372989 14832 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.374886 14829 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:34.375841 14829 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.376484 14826 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:34.376758 14824 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:34.377427 14826 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.377513 14824 tablet_bootstrap.cc:654] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.378270 14829 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:34.378393 14829 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:34.378538 14826 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:34.378640 14826 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:34.378779 14829 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.378882 14829 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.378921 14829 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379045 14829 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379145 14832 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:34.379217 14832 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:34.379331 14829 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.379376 14824 tablet_bootstrap.cc:492] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:34.379348 14826 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.379441 14824 ts_tablet_manager.cc:1367] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:34.379456 14826 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379485 14826 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379592 14826 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.379611 14832 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379695 14832 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379724 14832 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.379823 14832 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379818 14824 raft_consensus.cc:348] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.379946 14824 raft_consensus.cc:374] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.379981 14824 raft_consensus.cc:729] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.380091 14832 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.380107 14824 consensus_queue.cc:260] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.380369 14824 ts_tablet_manager.cc:1398] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.380375 14825 tablet_bootstrap.cc:492] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.380451 14825 ts_tablet_manager.cc:1367] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:34.380555 14826 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:34.380920 14825 raft_consensus.cc:348] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.381019 14825 raft_consensus.cc:374] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.381052 14825 raft_consensus.cc:729] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.381162 14825 consensus_queue.cc:260] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.381405 14825 ts_tablet_manager.cc:1398] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.381562 14825 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.382378 14825 tablet_bootstrap.cc:654] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.382462 14778 raft_consensus.cc:482] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.382537 14778 raft_consensus.cc:504] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.382861 14778 leader_election.cc:288] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:34.383679 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:34.383746 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:34.383790 8843 raft_consensus.cc:2454] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:34.384070 6496 leader_election.cc:302] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:34.384250 14778 raft_consensus.cc:2781] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 07:59:34.384266 6497 leader_election.cc:341] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:34.384294 14778 raft_consensus.cc:482] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.385007 14778 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.386086 14778 raft_consensus.cc:504] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.386334 14776 raft_consensus.cc:482] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.386404 14778 leader_election.cc:288] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:34.386418 14776 raft_consensus.cc:504] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.386732 14776 leader_election.cc:288] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:34.386791 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
W20240503 07:59:34.387066 6497 leader_election.cc:341] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:34.387137 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:34.387234 8975 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:34.387296 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:34.387394 6429 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:34.387405 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "757545bdc0e3426fa36c7ffcbd4189e1" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:34.387492 8843 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.387715 14825 tablet_bootstrap.cc:492] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.387791 14825 ts_tablet_manager.cc:1367] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:34.387768 6628 leader_election.cc:302] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:34.387955 14776 raft_consensus.cc:2781] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.387995 14776 raft_consensus.cc:482] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.388028 14776 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.388178 14825 raft_consensus.cc:348] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.388265 14825 raft_consensus.cc:374] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.388290 14825 raft_consensus.cc:729] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.388398 8843 raft_consensus.cc:2454] T 757545bdc0e3426fa36c7ffcbd4189e1 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:34.388394 14825 consensus_queue.cc:260] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.388649 14825 ts_tablet_manager.cc:1398] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:34.388859 6496 leader_election.cc:302] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters: d1f36d57661b4429a4be8d10b58d8ff7
I20240503 07:59:34.389009 14778 raft_consensus.cc:2781] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.389015 14776 raft_consensus.cc:504] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.389097 14778 raft_consensus.cc:686] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:34.389204 14778 consensus_queue.cc:237] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:34.389336 14776 leader_election.cc:288] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:34.389585 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:34.389675 8975 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.390412 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96720ba75f544ee18872449e7b26967f" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:34.390507 8975 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:34.390496 6429 raft_consensus.cc:3037] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.390763 6627 leader_election.cc:302] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.390924 14776 raft_consensus.cc:2781] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.391012 14776 raft_consensus.cc:686] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:34.391120 14776 consensus_queue.cc:237] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:34.391345 6429 raft_consensus.cc:2454] T 96720ba75f544ee18872449e7b26967f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:34.391604 6155 catalog_manager.cc:5484] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:34.392593 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.393870 6156 catalog_manager.cc:5484] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:34.403052 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.416043 14723 consensus_queue.cc:1035] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.418495 14723 consensus_queue.cc:1035] T 55ce814e0404442bbe40a36dfa3e5e30 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.427963 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:34.432687 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.432996 14821 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:34.434166 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.434458 14823 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:34.435932 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:34.436183 14822 tablet.cc:1620] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:34.438868 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 2)
I20240503 07:59:34.541458 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:34.546453 14821 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.547036 14823 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.547369 14822 tablet.cc:1722] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:34.552695 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00] alter complete (version 3)
I20240503 07:59:34.556813 14704 raft_consensus.cc:482] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:34.556911 14704 raft_consensus.cc:504] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.557301 14704 leader_election.cc:288] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:34.557590 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:34.557585 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:34.557703 8581 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:34.557703 6690 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:34.557943 8912 leader_election.cc:302] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.558112 14704 raft_consensus.cc:2781] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:34.558167 14704 raft_consensus.cc:482] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:34.558205 14704 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.559120 14704 raft_consensus.cc:504] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.559410 14704 leader_election.cc:288] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:34.559684 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:34.559695 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6246801a888a4ff3b280b812de0318e1" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:34.559783 8581 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.559779 6690 raft_consensus.cc:3037] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.560478 6690 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:34.560478 8581 raft_consensus.cc:2454] T 6246801a888a4ff3b280b812de0318e1 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:34.560742 8912 leader_election.cc:302] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:34.560914 14704 raft_consensus.cc:2781] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:34.561019 14704 raft_consensus.cc:686] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:34.561141 14704 consensus_queue.cc:237] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:34.563315 6148 catalog_manager.cc:5484] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:34.581804 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:34.591657 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:34.613097 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:34.654304 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.654436 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:34.656417 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.656987 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:34.657060 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:34.657332 14846 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:34.657449 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 4030f6767afe4113b35a1ad1fd668912 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:34 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:34.657471 14846 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:34.657660 14846 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.657763 14848 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:34.657924 14848 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.658046 14848 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.658185 14846 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.658898 14847 tablet_replica.cc:330] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:34.659094 14847 raft_consensus.cc:2229] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:34.659233 14847 raft_consensus.cc:2258] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:34.659775 14848 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.660676 14847 ts_tablet_manager.cc:1875] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:34.661324 14846 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.661394 14846 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.661609 14846 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:34.662153 14848 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.662225 14848 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.662360 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.662638 14848 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:34.662832 14847 ts_tablet_manager.cc:1888] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:34.662901 14847 log.cc:1196] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4030f6767afe4113b35a1ad1fd668912
I20240503 07:59:34.663180 14847 ts_tablet_manager.cc:1909] T 4030f6767afe4113b35a1ad1fd668912 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:34.663233 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.663705 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 4030f6767afe4113b35a1ad1fd668912 (table rebalancer_dynamic_table_new_name [id=326f2b2db40c4fff9898cff709bcbe00]) successfully deleted
I20240503 07:59:34.758107 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:34.763290 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763290 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763538 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.763538 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.763554 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 8726a50fd24d443e968cda5b39ae504f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]), partition=
I20240503 07:59:34.763792 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8726a50fd24d443e968cda5b39ae504f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:34.765764 14829 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:34.765874 14825 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:34.766592 14828 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:34.766656 14829 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.766732 14825 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.767406 14828 tablet_bootstrap.cc:654] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:34.767959 14829 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:34.768031 14829 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
W20240503 07:59:34.768453 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:34.768441 14829 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.768556 14829 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.768590 14829 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.768692 14829 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.768977 14829 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:34.769004 14828 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:34.769068 14828 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:34.769472 14828 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.769569 14828 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.769595 14828 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.769697 14828 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.769945 14828 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:34.771327 14825 tablet_bootstrap.cc:492] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:34.771425 14825 ts_tablet_manager.cc:1367] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:34.771842 14825 raft_consensus.cc:348] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.771947 14825 raft_consensus.cc:374] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:34.771981 14825 raft_consensus.cc:729] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:34.772078 14825 consensus_queue.cc:260] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:34.772351 14825 ts_tablet_manager.cc:1398] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:34.774981 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.790868 6429 raft_consensus.cc:3037] T 757545bdc0e3426fa36c7ffcbd4189e1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:34.792021 14356 consensus_queue.cc:1035] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.794730 14853 consensus_queue.cc:1035] T 757545bdc0e3426fa36c7ffcbd4189e1 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.884141 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:34.895361 14856 consensus_queue.cc:1035] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:34.899500 14856 consensus_queue.cc:1035] T 96720ba75f544ee18872449e7b26967f P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:34.998251 6366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 07:59:34.999013 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:35.006618 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.042444 14753 raft_consensus.cc:482] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.042547 14753 raft_consensus.cc:504] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.042918 14753 leader_election.cc:288] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.043258 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.043381 6559 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.043365 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.043476 8581 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.043736 6363 leader_election.cc:302] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.044102 14686 raft_consensus.cc:2781] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.044160 14686 raft_consensus.cc:482] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.044193 14686 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.045279 14686 raft_consensus.cc:504] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.045567 14686 leader_election.cc:288] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.045797 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.045853 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8726a50fd24d443e968cda5b39ae504f" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.045941 8581 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.045904 6559 raft_consensus.cc:3037] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.046887 6559 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.046898 8581 raft_consensus.cc:2454] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.047281 6363 leader_election.cc:302] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.047432 14753 raft_consensus.cc:2781] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.047533 14753 raft_consensus.cc:686] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:35.047642 14753 consensus_queue.cc:237] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.050334 6148 catalog_manager.cc:5484] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:35.051632 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:35.092316 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.103140 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.109166 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109287 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109475 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.109424 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 61e045ca076345339d21d95a4c79e769 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.109572 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.109666 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e045ca076345339d21d95a4c79e769. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.111629 14864 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.111634 14825 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.112449 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112557 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112600 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.112684 14825 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.112715 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.112828 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.112833 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.113464 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.113708 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.114984 14828 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.115417 14825 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.115492 14825 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.115746 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 8bee4ade303f4a93a8153f87fa8cf79b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.115916 14825 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.115983 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bee4ade303f4a93a8153f87fa8cf79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.116012 14825 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.116041 14828 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.116043 14825 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.116134 14868 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.116138 14825 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.116425 14825 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.117296 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 43c55a2d6fd84d84aac06b9d85065e7d (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=c9dbaa12419646eda826403fbae2f392]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.117422 14864 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.117518 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c55a2d6fd84d84aac06b9d85065e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.118041 14865 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.118551 14828 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.118634 14828 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.119082 14828 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.119182 14828 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.119220 14828 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.119366 14828 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.119400 14865 tablet_bootstrap.cc:654] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.119663 14828 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.119709 14825 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.120556 14825 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
W20240503 07:59:35.121336 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.121696 14825 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.121692 14865 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.121781 14825 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:35.121789 14865 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:35.122180 14825 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.122287 14825 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.122236 14865 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.122321 14825 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.122346 14865 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.122378 14865 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.122433 14825 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.122548 14865 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.122740 14825 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.122908 14865 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.123087 14865 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.124264 14865 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.125497 14865 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.125615 14865 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.126070 14865 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.126196 14865 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.126253 14865 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.126402 14865 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.126506 14867 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.126709 14865 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.127655 14867 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.128716 14868 tablet_bootstrap.cc:654] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.128751 14864 tablet_bootstrap.cc:492] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.128870 14864 ts_tablet_manager.cc:1367] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 07:59:35.129305 14864 raft_consensus.cc:348] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.129411 14864 raft_consensus.cc:374] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.129446 14864 raft_consensus.cc:729] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.129559 14864 consensus_queue.cc:260] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.129858 14864 ts_tablet_manager.cc:1398] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.130313 14868 tablet_bootstrap.cc:492] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.130347 14864 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.130393 14868 ts_tablet_manager.cc:1367] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:35.130805 14868 raft_consensus.cc:348] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.130906 14868 raft_consensus.cc:374] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.130939 14868 raft_consensus.cc:729] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
W20240503 07:59:35.130949 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.131047 14868 consensus_queue.cc:260] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.131228 14864 tablet_bootstrap.cc:654] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.131305 14868 ts_tablet_manager.cc:1398] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.132308 14864 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.132398 14864 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:35.132666 14867 tablet_bootstrap.cc:492] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.132742 14867 ts_tablet_manager.cc:1367] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:35.133131 14864 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133239 14864 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.133183 14867 raft_consensus.cc:348] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133267 14864 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.133275 14867 raft_consensus.cc:374] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.133306 14867 raft_consensus.cc:729] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.133407 14867 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133489 14864 consensus_queue.cc:260] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.133673 14867 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:35.133849 14864 ts_tablet_manager.cc:1398] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.134558 14806 raft_consensus.cc:482] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.134630 14806 raft_consensus.cc:504] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.134949 14806 leader_election.cc:288] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:35.135279 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:35.135386 6690 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:35.135689 6234 leader_election.cc:302] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:35.136171 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:35.136189 14806 raft_consensus.cc:2781] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.136252 14806 raft_consensus.cc:482] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.136291 14806 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.136278 6429 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:35.137348 14806 raft_consensus.cc:504] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.137710 14806 leader_election.cc:288] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:35.138113 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:35.138211 6429 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.138505 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e045ca076345339d21d95a4c79e769" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:35.138595 6690 raft_consensus.cc:3037] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.139081 6429 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:35.139487 6690 raft_consensus.cc:2454] T 61e045ca076345339d21d95a4c79e769 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
W20240503 07:59:35.139497 8779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 07:59:35.139720 6235 leader_election.cc:302] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.139890 14806 raft_consensus.cc:2781] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.139984 14806 raft_consensus.cc:686] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:35.140098 14806 consensus_queue.cc:237] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:35.142850 6150 catalog_manager.cc:5484] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:35.149704 8517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 07:59:35.159790 14844 consensus_queue.cc:1035] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.163247 14844 consensus_queue.cc:1035] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.163314 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:35.167068 14686 raft_consensus.cc:482] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.167157 14686 raft_consensus.cc:504] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.167600 14686 leader_election.cc:288] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:35.168085 8581 raft_consensus.cc:1261] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.168511 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:35.168643 8842 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:35.168726 6559 raft_consensus.cc:1261] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer d1f36d57661b4429a4be8d10b58d8ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.169099 14862 consensus_queue.cc:1035] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.169610 6365 leader_election.cc:302] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.169768 14862 raft_consensus.cc:2781] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.169809 14862 raft_consensus.cc:482] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.169857 14862 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.170912 14862 raft_consensus.cc:504] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.171250 14862 leader_election.cc:288] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:35.172029 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:35.172138 8842 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.173063 8842 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.173523 6365 leader_election.cc:302] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:35.173689 14753 consensus_queue.cc:1035] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.173960 14753 raft_consensus.cc:2781] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.174052 14753 raft_consensus.cc:686] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:35.174191 14753 consensus_queue.cc:237] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.174647 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
I20240503 07:59:35.176298 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.177543 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:35.177657 6298 raft_consensus.cc:3037] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.178359 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c55a2d6fd84d84aac06b9d85065e7d" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:35.178632 6298 raft_consensus.cc:2454] T 43c55a2d6fd84d84aac06b9d85065e7d P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:35.180114 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
W20240503 07:59:35.181529 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.182274 14886 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.185029 6148 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0]). Expected version 1 got 0
I20240503 07:59:35.185050 6155 catalog_manager.cc:5484] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.186969 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.187258 14885 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:35.188357 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.188820 14887 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.188998 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 1)
W20240503 07:59:35.196712 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.197345 6497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 07:59:35.197942 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.216382 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.229275 14859 raft_consensus.cc:482] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.229383 14859 raft_consensus.cc:504] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.229784 14859 leader_election.cc:288] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.230086 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.230207 6559 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.230279 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.230424 8975 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.230509 6625 leader_election.cc:302] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.230710 14859 raft_consensus.cc:2781] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.230758 14859 raft_consensus.cc:482] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.230799 14859 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.231601 14859 raft_consensus.cc:504] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.231956 14859 leader_election.cc:288] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.232163 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.232296 8975 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.233300 8975 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.233486 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bee4ade303f4a93a8153f87fa8cf79b" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.233580 6560 raft_consensus.cc:3037] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.234266 6560 raft_consensus.cc:2454] T 8bee4ade303f4a93a8153f87fa8cf79b P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.234464 6627 leader_election.cc:302] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:35.234942 14859 raft_consensus.cc:2781] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.235054 14859 raft_consensus.cc:686] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:35.235170 14859 consensus_queue.cc:237] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:35.237179 6150 catalog_manager.cc:5484] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.290171 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 07:59:35.293486 8913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 07:59:35.295421 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.295735 14885 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:35.295871 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.296124 14887 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.296979 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.297252 14886 tablet.cc:1620] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.301381 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 2)
W20240503 07:59:35.315912 6628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 07:59:35.341409 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:35.403426 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:35.408362 14885 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.408746 14887 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.408998 14886 tablet.cc:1722] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:35.414213 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0] alter complete (version 3)
W20240503 07:59:35.433310 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 07:59:35.516965 6155 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.517127 6155 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:35.519039 6155 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.519518 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:35.519631 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:35.519768 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8726a50fd24d443e968cda5b39ae504f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:35 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:35.519853 14889 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:35.519891 14890 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:35.519987 14889 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:35.520062 14891 tablet_replica.cc:330] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:35.520073 14890 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.520175 14889 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520187 14890 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520224 14891 raft_consensus.cc:2229] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:35.520352 14891 raft_consensus.cc:2258] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:35.520634 14889 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.520890 14891 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.520939 14890 ts_tablet_manager.cc:1875] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:35.522506 14889 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.522603 14889 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.522883 14889 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:35.522953 14891 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.523056 14891 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.523077 14890 ts_tablet_manager.cc:1888] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:35.523139 14890 log.cc:1196] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8726a50fd24d443e968cda5b39ae504f
I20240503 07:59:35.523363 14891 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:35.523407 14890 ts_tablet_manager.cc:1909] T 8726a50fd24d443e968cda5b39ae504f P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:35.523515 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
I20240503 07:59:35.523962 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
I20240503 07:59:35.524381 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 8726a50fd24d443e968cda5b39ae504f (table rebalancer_dynamic_table_new_name [id=aa0a95e2bf7247a08b58db4b17e6b1e0]) successfully deleted
W20240503 07:59:35.524906 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.527825 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:35.540796 6235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 07:59:35.542044 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 07:59:35.551419 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:35.607445 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.612282 14686 consensus_queue.cc:1035] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.620543 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:35.622504 14862 consensus_queue.cc:1035] T 43c55a2d6fd84d84aac06b9d85065e7d P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.630327 8955 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.630589 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.631083 8559 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.631311 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.633131 6540 tablet_service.cc:1467] Processing CreateTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]), partition=
I20240503 07:59:35.633322 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee56ad0414cc4138ad4802b44a2a3865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.634749 14867 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.637252 14896 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.638119 14896 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.639716 14897 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.640121 14867 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.644320 14897 tablet_bootstrap.cc:654] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.648283 14867 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.648370 14867 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:35.648403 14896 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.648471 14896 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:35.648826 14867 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.648885 14896 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.648938 14867 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.648967 14896 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.648967 14867 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.648995 14896 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.649098 14867 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.649405 14867 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.649523 14896 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.650015 14896 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.653101 14897 tablet_bootstrap.cc:492] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.653226 14897 ts_tablet_manager.cc:1367] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240503 07:59:35.653784 14897 raft_consensus.cc:348] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.653949 14897 raft_consensus.cc:374] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.654026 14897 raft_consensus.cc:729] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.654183 14897 consensus_queue.cc:260] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.656622 14897 ts_tablet_manager.cc:1398] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:35.669754 14759 raft_consensus.cc:482] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.669828 14759 raft_consensus.cc:504] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.670138 14759 leader_election.cc:288] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.670419 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.670506 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.670536 8976 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:35.670604 6559 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:35.670842 8516 leader_election.cc:302] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.671028 14759 raft_consensus.cc:2781] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.671079 14759 raft_consensus.cc:482] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.671115 14759 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.672271 14759 raft_consensus.cc:504] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.672598 14759 leader_election.cc:288] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:35.672938 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.673035 6559 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.673045 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee56ad0414cc4138ad4802b44a2a3865" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.673127 8976 raft_consensus.cc:3037] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.673890 6559 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:35.673920 8976 raft_consensus.cc:2454] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:35.674229 8516 leader_election.cc:302] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.674412 14759 raft_consensus.cc:2781] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.674446 14806 consensus_queue.cc:1035] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.674707 14759 raft_consensus.cc:686] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:35.674839 14759 consensus_queue.cc:237] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.677665 6150 catalog_manager.cc:5484] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:35.678550 14723 consensus_queue.cc:1035] T 61e045ca076345339d21d95a4c79e769 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.722406 14856 consensus_queue.cc:1035] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.726897 14856 consensus_queue.cc:1035] T 8bee4ade303f4a93a8153f87fa8cf79b P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.752368 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:35.758546 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.758837 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.759260 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.759374 6672 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.759505 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.759601 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.760962 14897 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:35.761639 14896 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.761862 14897 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.762271 14909 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:35.762559 14896 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.763211 6277 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763460 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.763463 8822 tablet_service.cc:1467] Processing CreateTablet for tablet 63933b2ad5fc442e936a8cb4c273c30c (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.763718 8822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63933b2ad5fc442e936a8cb4c273c30c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766091 14909 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.766279 8955 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.766520 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.766839 14897 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:35.766911 14897 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 07:59:35.767153 14909 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:35.767222 14909 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:35.767316 14897 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767436 14897 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.767467 14897 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.767585 14897 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767644 14909 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.767720 14909 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.767748 14909 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.767853 14897 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 07:59:35.767858 14909 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.768105 14909 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.769163 14867 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:35.769935 14867 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.763463 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.771070 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.772097 14867 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:35.772176 14867 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.772528 14867 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.772611 14867 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.772641 14867 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.772748 14867 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.773042 14867 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.774139 6408 tablet_service.cc:1467] Processing CreateTablet for tablet d3cadfbbb21d44cd95f2132e964e7715 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.774398 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3cadfbbb21d44cd95f2132e964e7715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.776183 14916 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:35.777201 8559 tablet_service.cc:1467] Processing CreateTablet for tablet cf58de4ceb76453bb7032853cdd91bdc (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=a99f1a7ed8be4724a77753d546ea1826]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:35.777282 14916 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.777422 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf58de4ceb76453bb7032853cdd91bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:35.778316 14916 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:35.778393 14916 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:35.778889 14916 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.778987 14916 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.779018 14916 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.779129 14916 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.779398 14916 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.779719 14913 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.780576 14913 tablet_bootstrap.cc:654] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.781215 14896 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.781311 14896 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s
I20240503 07:59:35.781692 14896 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.781783 14896 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.781811 14896 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.781908 14896 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.782164 14896 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:35.782274 14896 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:35.782878 14896 tablet_bootstrap.cc:654] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.784507 14896 tablet_bootstrap.cc:492] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:35.784579 14896 ts_tablet_manager.cc:1367] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:35.785468 14896 raft_consensus.cc:348] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.785553 14896 raft_consensus.cc:374] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.785575 14896 raft_consensus.cc:729] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.785681 14896 consensus_queue.cc:260] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.785952 14896 ts_tablet_manager.cc:1398] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:35.785964 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:35.786610 14912 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:35.787510 14912 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.788301 14913 tablet_bootstrap.cc:492] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.788385 14913 ts_tablet_manager.cc:1367] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:35.788715 14913 raft_consensus.cc:348] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.788818 14913 raft_consensus.cc:374] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.788851 14912 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:35.788872 14913 raft_consensus.cc:729] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.788971 14913 consensus_queue.cc:260] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.789155 14912 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:35.789314 14913 ts_tablet_manager.cc:1398] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.789693 14912 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.789814 14912 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.789916 14912 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.790082 14913 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:35.790052 14912 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.790498 14912 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:35.790921 14913 tablet_bootstrap.cc:654] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:35.792490 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:35.793534 14913 tablet_bootstrap.cc:492] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:35.793597 14913 ts_tablet_manager.cc:1367] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:35.793973 14913 raft_consensus.cc:348] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.794137 14913 raft_consensus.cc:374] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:35.794211 14913 raft_consensus.cc:729] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:35.794373 14913 consensus_queue.cc:260] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:35.796753 14913 ts_tablet_manager.cc:1398] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:35.797502 8976 raft_consensus.cc:1261] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.797569 6559 raft_consensus.cc:1261] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:35.797962 14843 consensus_queue.cc:1035] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.798722 14843 consensus_queue.cc:1035] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:35.801429 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.801702 14887 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.802111 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.803936 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.804179 14886 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.804085 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.804697 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:35.804980 14924 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:35.807441 6150 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]). Expected version 1 got 0
I20240503 07:59:35.808920 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 1)
I20240503 07:59:35.826041 14667 raft_consensus.cc:482] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.826125 14667 raft_consensus.cc:504] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.826467 14667 leader_election.cc:288] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.826735 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.826728 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:35.826849 8575 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:35.826855 6559 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:35.827100 8776 leader_election.cc:302] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.827260 14667 raft_consensus.cc:2781] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.827313 14667 raft_consensus.cc:482] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.827366 14667 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.828007 14667 raft_consensus.cc:504] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.828276 14667 leader_election.cc:288] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.828615 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:35.828614 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63933b2ad5fc442e936a8cb4c273c30c" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.828711 8575 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.828711 6559 raft_consensus.cc:3037] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.829558 8575 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:35.829558 6559 raft_consensus.cc:2454] T 63933b2ad5fc442e936a8cb4c273c30c P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:35.829821 8776 leader_election.cc:302] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:35.830085 14667 raft_consensus.cc:2781] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.830171 14667 raft_consensus.cc:686] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:35.830310 14667 consensus_queue.cc:237] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.832265 6155 catalog_manager.cc:5484] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.874724 14427 raft_consensus.cc:482] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:35.874810 14427 raft_consensus.cc:504] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.875092 14427 leader_election.cc:288] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.875361 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:35.875399 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:35.875500 8575 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.875499 8976 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:35.875742 6625 leader_election.cc:302] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.875898 14427 raft_consensus.cc:2781] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:35.875959 14427 raft_consensus.cc:482] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:35.876003 14427 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.876866 14427 raft_consensus.cc:504] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.877147 14427 leader_election.cc:288] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:35.877434 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:35.877522 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf58de4ceb76453bb7032853cdd91bdc" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:35.877560 8976 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.877600 8575 raft_consensus.cc:3037] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:35.878216 8575 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.878330 8976 raft_consensus.cc:2454] T cf58de4ceb76453bb7032853cdd91bdc P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:35.878479 6625 leader_election.cc:302] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:35.878667 14427 raft_consensus.cc:2781] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:35.878744 14427 raft_consensus.cc:686] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:35.878860 14427 consensus_queue.cc:237] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:35.880817 6155 catalog_manager.cc:5484] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:35.912032 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:35.917513 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.917948 14887 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:35.918627 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.918939 14924 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:35.918977 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:35.919255 14886 tablet.cc:1620] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:35.924317 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 2)
W20240503 07:59:36.024842 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.025696 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:36.035336 14887 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.038213 14924 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.038975 14886 tablet.cc:1722] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.041217 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7] alter complete (version 3)
W20240503 07:59:36.075618 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.145476 6148 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.145614 6148 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.147480 6148 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.147992 8955 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:36.148352 14929 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:36.148334 8559 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:36.148466 6540 tablet_service.cc:1514] Processing DeleteTablet for tablet ee56ad0414cc4138ad4802b44a2a3865 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:36.148540 14929 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.148648 14929 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.148761 14931 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:36.148943 14931 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.149202 14930 tablet_replica.cc:330] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:36.149363 14930 raft_consensus.cc:2229] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.149439 14929 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.149576 14930 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.150146 14930 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.151319 14929 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.151381 14929 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.151618 14929 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:36.152299 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.152338 14930 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.152402 14930 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.152664 14930 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:36.149114 14931 raft_consensus.cc:2258] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.153821 14931 ts_tablet_manager.cc:1875] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.154199 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.155555 14931 ts_tablet_manager.cc:1888] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.155614 14931 log.cc:1196] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee56ad0414cc4138ad4802b44a2a3865
I20240503 07:59:36.155839 14931 ts_tablet_manager.cc:1909] T ee56ad0414cc4138ad4802b44a2a3865 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:36.156383 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet ee56ad0414cc4138ad4802b44a2a3865 (table rebalancer_dynamic_table_new_name [id=ae66a17f2a0a42f4bc5ad9a9aede23c7]) successfully deleted
I20240503 07:59:36.220166 14667 consensus_queue.cc:1035] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:36.222419 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.223193 14905 consensus_queue.cc:1035] T 63933b2ad5fc442e936a8cb4c273c30c P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.232893 14753 raft_consensus.cc:482] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.232981 14753 raft_consensus.cc:504] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.233266 14753 leader_election.cc:288] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:36.233541 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:36.233633 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:36.233664 8843 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.233740 6298 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.233922 6365 leader_election.cc:302] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.234076 14753 raft_consensus.cc:2781] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.234130 14753 raft_consensus.cc:482] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.234155 14753 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.234911 14753 raft_consensus.cc:504] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.235234 14753 leader_election.cc:288] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:36.235529 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:36.235632 6298 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.235606 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3cadfbbb21d44cd95f2132e964e7715" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:36.235705 8843 raft_consensus.cc:3037] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.236397 6298 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.236419 8843 raft_consensus.cc:2454] T d3cadfbbb21d44cd95f2132e964e7715 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.236671 6363 leader_election.cc:302] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.236866 14753 raft_consensus.cc:2781] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.237010 14753 raft_consensus.cc:686] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:36.237136 14753 consensus_queue.cc:237] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:36.239583 6150 catalog_manager.cc:5484] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:36.248960 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:36.258143 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.258179 8559 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.258460 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.258522 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.260622 14896 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.260717 14909 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:36.261536 14896 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.261677 14909 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.262662 14896 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.262698 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 47173340c9c04be89fca6bf868782b68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]), partition=
I20240503 07:59:36.262735 14896 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.262928 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47173340c9c04be89fca6bf868782b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.263311 14896 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.263414 14896 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.263443 14896 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.263537 14896 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.263792 14896 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.265224 14912 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:36.266201 14912 tablet_bootstrap.cc:654] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.267060 14909 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:36.267158 14909 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:36.267550 14909 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.267666 14909 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.267704 14909 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.267869 14909 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.268165 14909 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.269520 14912 tablet_bootstrap.cc:492] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:36.269611 14912 ts_tablet_manager.cc:1367] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:36.270082 14912 raft_consensus.cc:348] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.270186 14912 raft_consensus.cc:374] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.270221 14912 raft_consensus.cc:729] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.270334 14912 consensus_queue.cc:260] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.270643 14912 ts_tablet_manager.cc:1398] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.353006 14859 raft_consensus.cc:482] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.353101 14859 raft_consensus.cc:504] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.353415 14859 leader_election.cc:288] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:36.353696 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:36.353737 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:36.353816 8575 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:36.353839 6298 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:36.354089 6625 leader_election.cc:302] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:36.354256 14859 raft_consensus.cc:2781] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.354344 14859 raft_consensus.cc:482] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.354393 14859 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.355419 14859 raft_consensus.cc:504] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.355731 14859 leader_election.cc:288] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:36.356087 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:36.356206 8575 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.356209 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47173340c9c04be89fca6bf868782b68" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:36.356304 6298 raft_consensus.cc:3037] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.357100 8575 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:36.357283 6298 raft_consensus.cc:2454] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:36.357445 6625 leader_election.cc:302] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:36.357604 14859 raft_consensus.cc:2781] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.357708 14859 raft_consensus.cc:686] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:36.357998 14859 consensus_queue.cc:237] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:36.359988 6150 catalog_manager.cc:5484] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:36.407457 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:36.411500 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:36.419432 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.439360 14859 consensus_queue.cc:1035] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.444350 14859 consensus_queue.cc:1035] T cf58de4ceb76453bb7032853cdd91bdc P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.484067 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:36.488139 6298 raft_consensus.cc:1261] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.488488 8581 raft_consensus.cc:1261] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:36.488592 14859 consensus_queue.cc:1035] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.489195 14427 consensus_queue.cc:1035] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.492120 6148 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.492792 6150 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.493794 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.494159 14950 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:36.494869 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.495170 14949 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:36.495290 6150 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.496480 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:36.496803 14887 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:36.497359 6150 catalog_manager.cc:5576] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.500597 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88]). Expected version 1 got 0
I20240503 07:59:36.500749 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 1)
W20240503 07:59:36.520156 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:36.521711 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.604854 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:36.611068 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.611411 14950 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: MemRowSet was empty: no flush needed.
I20240503 07:59:36.610810 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.611667 14949 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:36.612879 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:36.613158 14887 tablet.cc:1620] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:36.618748 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 2)
W20240503 07:59:36.682842 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.701711 14753 consensus_queue.cc:1035] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:36.704981 14928 consensus_queue.cc:1035] T d3cadfbbb21d44cd95f2132e964e7715 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:36.710593 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.733937 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:36.738978 14949 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.740032 14950 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.743027 14887 tablet.cc:1722] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:36.744103 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88] alter complete (version 3)
W20240503 07:59:36.834723 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:36.840832 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:36.845472 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.845588 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:36.847157 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.847494 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.847822 6408 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.848104 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.848529 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.848819 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.849880 14956 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:36.850195 14957 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:36.850307 8559 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.850631 8559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.851207 14957 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.851249 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.851483 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.851619 14956 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.852730 14961 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.852908 14956 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:36.852890 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.852979 14956 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.853106 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.853400 14956 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.853487 14956 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.853514 14956 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.853641 14956 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.853731 14961 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.853891 14956 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.854106 6540 tablet_service.cc:1467] Processing CreateTablet for tablet 7e65ff59abcc4d2ca815db3c544cf2b4 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.854374 6540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e65ff59abcc4d2ca815db3c544cf2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.854859 14961 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.854984 14961 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.854979 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 7c281d57b040492089ef29c3110fdf98 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.855154 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c281d57b040492089ef29c3110fdf98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.855437 14961 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.855532 14961 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.855564 14961 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.855696 14961 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.856789 6541 tablet_service.cc:1467] Processing CreateTablet for tablet c894ca4882b94f2ea07a0b59adc47ce7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=02b9b371d2fd4ffc9e5588c2b5010632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:36.856886 14963 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.857023 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c894ca4882b94f2ea07a0b59adc47ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.857412 14961 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.857579 14961 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:36.859055 14961 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.860003 14963 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.861421 14961 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:36.861467 14958 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:36.861513 14961 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:36.862277 14958 tablet_bootstrap.cc:654] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.862304 14961 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.862481 14961 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.862560 14961 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.862747 14961 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.863250 14958 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:36.863366 14958 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.863687 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.863840 14958 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.864023 14958 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.864092 14958 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.864110 14961 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:36.864274 14958 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.864506 6277 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:48112
I20240503 07:59:36.864969 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:36.864961 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 47173340c9c04be89fca6bf868782b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:36 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:36.865097 14967 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: stopping tablet replica
I20240503 07:59:36.865240 14967 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.865293 14958 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.865483 14967 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.866384 14958 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:36.867481 14958 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.867830 14969 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:36.868029 14969 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:36.868289 14969 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.868412 14967 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.868598 14968 tablet_replica.cc:330] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:36.869030 14969 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.871671 14967 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.871673 14968 raft_consensus.cc:2229] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:36.871755 14967 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.871806 14968 raft_consensus.cc:2258] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:36.872119 14967 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P c167a0a433d34b74affed13cbf96e302: Deleting consensus metadata
I20240503 07:59:36.872381 14969 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.872865 14968 ts_tablet_manager.cc:1875] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:36.872874 6141 catalog_manager.cc:4830] TS c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.873414 14958 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:36.873543 14958 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:36.874018 14958 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.874114 14958 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.874142 14958 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.874265 14958 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.874497 14969 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.874531 14958 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.874795 14969 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:36.875413 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.875792 14968 ts_tablet_manager.cc:1888] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:36.875859 14968 log.cc:1196] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/47173340c9c04be89fca6bf868782b68
I20240503 07:59:36.876065 14968 ts_tablet_manager.cc:1909] T 47173340c9c04be89fca6bf868782b68 P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:36.876691 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 47173340c9c04be89fca6bf868782b68 (table rebalancer_dynamic_table_new_name [id=a0ee631440d64b18ac6ed6d1bf44ea88]) successfully deleted
I20240503 07:59:36.877244 14957 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:36.877350 14957 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 07:59:36.877511 14963 tablet_bootstrap.cc:492] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.877606 14963 ts_tablet_manager.cc:1367] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s
I20240503 07:59:36.877861 14957 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.877971 14957 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.878005 14957 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.878139 14957 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.878430 14957 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:36.878561 14963 raft_consensus.cc:348] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.878666 14963 raft_consensus.cc:374] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.878698 14963 raft_consensus.cc:729] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.878813 14963 consensus_queue.cc:260] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:36.879088 14963 ts_tablet_manager.cc:1398] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 07:59:36.879259 14963 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.880139 14963 tablet_bootstrap.cc:654] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.884250 14963 tablet_bootstrap.cc:492] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.884323 14963 ts_tablet_manager.cc:1367] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 07:59:36.884655 14963 raft_consensus.cc:348] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.884739 14963 raft_consensus.cc:374] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.884794 14963 raft_consensus.cc:729] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.884939 14963 consensus_queue.cc:260] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:36.885262 14963 ts_tablet_manager.cc:1398] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.885424 14963 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:36.886288 14963 tablet_bootstrap.cc:654] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.887418 14963 tablet_bootstrap.cc:492] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.887513 14963 ts_tablet_manager.cc:1367] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:36.887925 14963 raft_consensus.cc:348] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.888043 14963 raft_consensus.cc:374] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.888083 14963 raft_consensus.cc:729] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.888196 14963 consensus_queue.cc:260] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.888478 14963 ts_tablet_manager.cc:1398] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240503 07:59:36.946274 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:36.959203 14928 raft_consensus.cc:482] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:36.959296 14928 raft_consensus.cc:504] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.959614 14928 leader_election.cc:288] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:36.959929 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:36.959952 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:36.960052 8581 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.960054 6560 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:36.960286 6363 leader_election.cc:302] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.960453 14928 raft_consensus.cc:2781] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:36.960506 14928 raft_consensus.cc:482] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:36.960530 14928 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.961421 14928 raft_consensus.cc:504] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.961701 14928 leader_election.cc:288] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:36.961988 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:36.962015 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c894ca4882b94f2ea07a0b59adc47ce7" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:36.962091 6560 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.962100 8581 raft_consensus.cc:3037] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:36.962814 8581 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.962941 6560 raft_consensus.cc:2454] T c894ca4882b94f2ea07a0b59adc47ce7 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:36.963131 6363 leader_election.cc:302] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:36.963279 14928 raft_consensus.cc:2781] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:36.963363 14928 raft_consensus.cc:686] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:36.963500 14928 consensus_queue.cc:237] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:36.965371 6148 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:36.966090 6156 catalog_manager.cc:5484] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:36.970693 6366 consensus_peers.cc:485] T 3488a545a7d84e389c5894d9ba3a592b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:36.971555 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.971797 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.971778 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.972010 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.973922 14956 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:36.974504 6541 tablet_service.cc:1467] Processing CreateTablet for tablet 724b4aae3de544aabff9ab7f790c5283 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]), partition=
I20240503 07:59:36.974685 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724b4aae3de544aabff9ab7f790c5283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:36.974815 14956 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.976218 14956 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:36.976302 14956 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:36.976466 14963 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
W20240503 07:59:36.976994 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:36.976749 14956 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.977221 14974 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:36.977244 14956 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.977280 14956 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.977355 14963 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.977387 14956 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.977664 14956 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:36.978129 14974 tablet_bootstrap.cc:654] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:36.979707 14963 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:36.979779 14963 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:36.980525 14963 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.980612 14963 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.980638 14963 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.980746 14963 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.981029 14963 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:36.982335 14974 tablet_bootstrap.cc:492] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:36.982446 14974 ts_tablet_manager.cc:1367] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:36.982875 14974 raft_consensus.cc:348] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.983018 14974 raft_consensus.cc:374] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:36.983068 14974 raft_consensus.cc:729] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:36.983251 14974 consensus_queue.cc:260] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:36.983577 14974 ts_tablet_manager.cc:1398] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:36.984396 6144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 07:59:36.985538 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd4400894d14f839670d209468f1929: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:37.011873 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.016990 14853 raft_consensus.cc:482] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.017083 14853 raft_consensus.cc:504] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.017464 14853 leader_election.cc:288] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:37.017741 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.017835 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:37.017871 8843 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:37.017962 8975 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:37.018126 6496 leader_election.cc:302] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.018291 14853 raft_consensus.cc:2781] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.018348 14853 raft_consensus.cc:482] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.018390 14853 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.019426 14853 raft_consensus.cc:504] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.019737 14853 leader_election.cc:288] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089)
I20240503 07:59:37.020032 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:37.020073 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e65ff59abcc4d2ca815db3c544cf2b4" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.020123 8975 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.020179 8843 raft_consensus.cc:3037] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.020973 8843 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:37.021077 8975 raft_consensus.cc:2454] T 7e65ff59abcc4d2ca815db3c544cf2b4 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:37.021229 6496 leader_election.cc:302] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.021509 14853 raft_consensus.cc:2781] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.021601 14853 raft_consensus.cc:686] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:37.021734 14853 consensus_queue.cc:237] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } }
I20240503 07:59:37.024148 6150 catalog_manager.cc:5484] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.035825 14844 raft_consensus.cc:482] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.035919 14844 raft_consensus.cc:504] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.036337 14844 leader_election.cc:288] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.036702 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.036792 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:37.036849 6560 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.036900 8581 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.037096 8910 leader_election.cc:302] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.037241 14844 raft_consensus.cc:2781] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.037300 14844 raft_consensus.cc:482] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.037335 14844 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.038950 14844 raft_consensus.cc:504] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.039278 14844 leader_election.cc:288] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.039502 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.039538 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c281d57b040492089ef29c3110fdf98" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:37.039609 6560 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.039616 8581 raft_consensus.cc:3037] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.040495 8581 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.040495 6560 raft_consensus.cc:2454] T 7c281d57b040492089ef29c3110fdf98 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.040812 8910 leader_election.cc:302] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.041066 14844 raft_consensus.cc:2781] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.041149 14844 raft_consensus.cc:686] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:37.041270 14844 consensus_queue.cc:237] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.043865 6150 catalog_manager.cc:5484] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:37.053076 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.109584 6235 consensus_peers.cc:485] T 6f260c4c4e7e4fcc8fbce11064bdc338 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:37.213280 14427 raft_consensus.cc:482] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.213380 14427 raft_consensus.cc:504] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.213701 14427 leader_election.cc:288] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:37.213946 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.213999 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.214077 8843 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:37.214135 6560 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:37.214388 6625 leader_election.cc:302] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.214561 14427 raft_consensus.cc:2781] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.214619 14427 raft_consensus.cc:482] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.214655 14427 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.215875 14427 raft_consensus.cc:504] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.216163 14427 leader_election.cc:288] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:37.216424 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.216477 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724b4aae3de544aabff9ab7f790c5283" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.216544 8843 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.216565 6560 raft_consensus.cc:3037] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.217495 6560 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:37.217610 8843 raft_consensus.cc:2454] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:37.217808 6625 leader_election.cc:302] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.218029 14427 raft_consensus.cc:2781] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.218147 14427 raft_consensus.cc:686] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:37.218291 14427 consensus_queue.cc:237] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.220551 6150 catalog_manager.cc:5484] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:37.282526 8517 consensus_peers.cc:485] T f635e30fb6974fa1a79d8a46eecc3d5e P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:37.372349 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:37.376359 8842 raft_consensus.cc:1261] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.376720 14427 consensus_queue.cc:1035] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.376888 6560 raft_consensus.cc:1261] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer a97073f89fd44702bccb74a9b76f858d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:37.377306 14427 consensus_queue.cc:1035] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.380676 6150 catalog_manager.cc:5576] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.381660 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.383313 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.383674 14991 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:37.385409 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.385527 6156 catalog_manager.cc:5576] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.385903 14993 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:37.386121 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5]). Expected version 1 got 0
I20240503 07:59:37.389418 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:37.389775 14992 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.390570 14984 consensus_queue.cc:1035] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.391659 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 1)
W20240503 07:59:37.394142 8913 consensus_peers.cc:485] T e12d0243dbb54bdfa1f584011a3db5b6 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.396624 14928 consensus_queue.cc:1035] T c894ca4882b94f2ea07a0b59adc47ce7 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:37.412698 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:37.470705 14844 consensus_queue.cc:1035] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.473464 14844 consensus_queue.cc:1035] T 7c281d57b040492089ef29c3110fdf98 P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.474686 14853 consensus_queue.cc:1035] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.477552 14853 consensus_queue.cc:1035] T 7e65ff59abcc4d2ca815db3c544cf2b4 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:37.494136 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:37.498935 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.499291 14991 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: MemRowSet was empty: no flush needed.
I20240503 07:59:37.499310 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.499575 14992 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:37.499946 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:37.500252 14993 tablet.cc:1620] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: MemRowSet was empty: no flush needed.
I20240503 07:59:37.506247 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 2)
W20240503 07:59:37.530040 8517 consensus_peers.cc:485] T 7995f12a988548088d45dc4d7a1b9aa0 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.548743 6497 consensus_peers.cc:485] T 3e6eef6b9c8548c3a790d42a6a9eef4c P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.555933 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240503 07:59:37.559702 6366 consensus_peers.cc:485] T 8384cbf3f368424d9f8a9060efe9c0f2 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.561957 6277 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562120 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562260 6277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.562408 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.562652 6279 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.562891 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.564646 15007 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:37.565236 15006 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.565918 8824 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.565997 15007 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.565996 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.566159 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.566258 15006 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.566269 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.567631 15006 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.567723 15006 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.567981 15010 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:37.568192 15006 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.568301 15006 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.568336 15006 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.568461 15006 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.568795 15006 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 07:59:37.568625 15011 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:37.569139 15006 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.569202 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d74b95ba4bf64088898639dcd96f5730 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569244 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 965f6a33bacb4776b59360d66a6d4cb1 (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569451 6410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74b95ba4bf64088898639dcd96f5730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.569696 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.569849 15011 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.569921 6409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.570029 15006 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.571517 15010 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.571861 15006 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.565189 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 01384cbe92504a839474a60048fe293e (DEFAULT_TABLE table=rebalancer_extra_table_70 [id=58e7e2d7030648529062bace7a4c31a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:37.571991 15006 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.572125 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01384cbe92504a839474a60048fe293e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.572423 15006 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.572503 15006 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.572508 15010 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:37.572535 15006 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.572582 15010 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:37.572916 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f6a33bacb4776b59360d66a6d4cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.573040 15014 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.572661 15006 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.573698 15010 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.573791 15010 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.573896 15010 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.573983 15006 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:37.574035 15010 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.574295 15010 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:37.575001 15011 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:37.575096 15011 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240503 07:59:37.575352 15014 tablet_bootstrap.cc:654] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.575508 15011 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.575595 15011 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.575623 15011 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.575748 15011 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.576040 15011 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.579473 15007 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:37.579474 15014 tablet_bootstrap.cc:492] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.579555 15014 ts_tablet_manager.cc:1367] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:37.579564 15007 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:37.579939 15014 raft_consensus.cc:348] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.580034 15014 raft_consensus.cc:374] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.580060 15014 raft_consensus.cc:729] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.580045 15007 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.580143 15007 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.580188 15007 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.580158 15014 consensus_queue.cc:260] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.580287 15006 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:37.580307 15007 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.580402 15014 ts_tablet_manager.cc:1398] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.580572 15007 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.581445 15014 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.581965 15006 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.582577 15014 tablet_bootstrap.cc:654] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.583561 15006 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:37.583827 15006 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:37.584276 15006 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.584429 15006 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.584481 15006 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.584587 15006 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.584867 15006 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:37.587340 15014 tablet_bootstrap.cc:492] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.587420 15014 ts_tablet_manager.cc:1367] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:37.587812 15014 raft_consensus.cc:348] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.587898 15014 raft_consensus.cc:374] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.587924 15014 raft_consensus.cc:729] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.588016 15014 consensus_queue.cc:260] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.588256 15014 ts_tablet_manager.cc:1398] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:37.588375 15014 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:37.589228 15014 tablet_bootstrap.cc:654] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.590948 15014 tablet_bootstrap.cc:492] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:37.591030 15014 ts_tablet_manager.cc:1367] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:37.591380 15014 raft_consensus.cc:348] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.591452 15014 raft_consensus.cc:374] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.591480 15014 raft_consensus.cc:729] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.591583 15014 consensus_queue.cc:260] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.591832 15014 ts_tablet_manager.cc:1398] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:37.607347 6156 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:37.611970 14991 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.612707 14992 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.615487 14993 tablet.cc:1722] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:37.617928 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5] alter complete (version 3)
W20240503 07:59:37.623849 8517 consensus_peers.cc:485] T cd978026a0aa42758f07a701131fdc27 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:37.624459 6366 consensus_peers.cc:485] T 974c815ff8e94190a7bf28444cb3cd9f P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.678319 14985 raft_consensus.cc:482] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.678444 14985 raft_consensus.cc:504] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.678822 14985 leader_election.cc:288] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:37.679131 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:37.679152 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d" is_pre_election: true
I20240503 07:59:37.679261 6429 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:37.679286 6690 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:37.679581 6234 leader_election.cc:302] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:37.679730 14985 raft_consensus.cc:2781] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.679787 14985 raft_consensus.cc:482] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.679828 14985 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.680670 14985 raft_consensus.cc:504] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.681013 14985 leader_election.cc:288] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651), a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733)
I20240503 07:59:37.681303 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:37.681326 6690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f6a33bacb4776b59360d66a6d4cb1" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97073f89fd44702bccb74a9b76f858d"
I20240503 07:59:37.681418 6429 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.681444 6690 raft_consensus.cc:3037] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.682374 6429 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:37.682386 6690 raft_consensus.cc:2454] T 965f6a33bacb4776b59360d66a6d4cb1 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:37.682641 6235 leader_election.cc:302] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.682785 14985 raft_consensus.cc:2781] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.682860 14985 raft_consensus.cc:686] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:37.682989 14985 consensus_queue.cc:237] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:37.685518 6150 catalog_manager.cc:5484] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:37.722133 6150 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.722296 6150 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:37.724279 6150 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.724751 8824 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:37500
I20240503 07:59:37.724982 6672 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:38718
I20240503 07:59:37.725040 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet 724b4aae3de544aabff9ab7f790c5283 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:37 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:37.725265 15022 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: stopping tablet replica
I20240503 07:59:37.725329 15021 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: stopping tablet replica
I20240503 07:59:37.725346 15023 tablet_replica.cc:330] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:37.725453 15022 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:37.725463 15023 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.725574 15023 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.725559 15021 raft_consensus.cc:2229] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:37.725646 15022 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.725920 15021 raft_consensus.cc:2258] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:37.726135 15023 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.726135 15022 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.726416 15021 ts_tablet_manager.cc:1875] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:37.728338 15023 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728340 15022 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728387 15021 ts_tablet_manager.cc:1888] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:37.728412 15022 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728477 15023 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728489 15021 log.cc:1196] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/724b4aae3de544aabff9ab7f790c5283
I20240503 07:59:37.728787 15021 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P 616fdda6ba714cb7be01a564de2b9db2: Deleting consensus metadata
I20240503 07:59:37.728871 15023 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:37.728997 15022 ts_tablet_manager.cc:1909] T 724b4aae3de544aabff9ab7f790c5283 P a97073f89fd44702bccb74a9b76f858d: Deleting consensus metadata
I20240503 07:59:37.729477 6143 catalog_manager.cc:4830] TS 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.729523 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.729918 6143 catalog_manager.cc:4830] TS a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): tablet 724b4aae3de544aabff9ab7f790c5283 (table rebalancer_dynamic_table_new_name [id=1924fd3110e44015bb6979cd4b978ad5]) successfully deleted
I20240503 07:59:37.736845 14952 raft_consensus.cc:482] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.736934 14952 raft_consensus.cc:504] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.737277 14952 leader_election.cc:288] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:37.737533 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:37.737668 6296 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.737672 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:37.737782 8843 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.737996 6363 leader_election.cc:302] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.738148 14952 raft_consensus.cc:2781] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.738204 14952 raft_consensus.cc:482] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.738240 14952 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.739117 14952 raft_consensus.cc:504] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.739456 14952 leader_election.cc:288] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417)
I20240503 07:59:37.739725 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:37.739743 8842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74b95ba4bf64088898639dcd96f5730" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:37.739825 6296 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.739838 8842 raft_consensus.cc:3037] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.740715 8842 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.740854 6296 raft_consensus.cc:2454] T d74b95ba4bf64088898639dcd96f5730 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.741096 6363 leader_election.cc:302] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.741309 14952 raft_consensus.cc:2781] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.741395 14952 raft_consensus.cc:686] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:37.741521 14952 consensus_queue.cc:237] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.743726 6150 catalog_manager.cc:5484] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:37.767725 14984 raft_consensus.cc:482] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.767848 14984 raft_consensus.cc:504] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.768150 14984 leader_election.cc:288] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:37.768448 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:37.768473 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:37.768559 6296 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.768590 8976 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 0.
I20240503 07:59:37.768824 6363 leader_election.cc:302] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.768977 14984 raft_consensus.cc:2781] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.769034 14984 raft_consensus.cc:482] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.769078 14984 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.769915 14984 raft_consensus.cc:504] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.770202 14984 leader_election.cc:288] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:37.770418 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:37.770527 8976 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.770586 6296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01384cbe92504a839474a60048fe293e" candidate_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:37.770689 6296 raft_consensus.cc:3037] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.771250 8976 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.771368 6296 raft_consensus.cc:2454] T 01384cbe92504a839474a60048fe293e P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f36d57661b4429a4be8d10b58d8ff7 in term 1.
I20240503 07:59:37.771613 6363 leader_election.cc:302] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:37.771754 14984 raft_consensus.cc:2781] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.771835 14984 raft_consensus.cc:686] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Running, Role: LEADER
I20240503 07:59:37.771955 14984 consensus_queue.cc:237] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:37.774163 6150 catalog_manager.cc:5484] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:37.814859 6235 consensus_peers.cc:485] T 7f417c61594348428ee9514ded49ee88 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.826284 6150 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:37.832142 8956 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.832398 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.834615 15011 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:37.834957 6541 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.835458 15011 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.835199 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.835659 8560 tablet_service.cc:1467] Processing CreateTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]), partition=
I20240503 07:59:37.835884 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91cdfb905944c1cbed56ab5cbd1253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:37.836481 15011 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:37.836565 15011 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:37.836983 15011 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.837074 15011 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.837102 15011 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.837204 15011 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.837447 15011 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:37.841234 6235 consensus_peers.cc:485] T bfd4259777e5492c904e6d65a36d92ed P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:37.841384 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:37.841729 15027 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:37.842762 15028 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:37.842917 15027 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.843640 15028 tablet_bootstrap.cc:654] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:37.844048 15027 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:37.844147 15027 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:37.844637 15027 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.844744 15027 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.844797 15027 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.844938 15027 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.845265 15027 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 07:59:37.846403 8779 consensus_peers.cc:485] T cfd4deb6a5104141bfa51fe4114bb708 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.846562 15028 tablet_bootstrap.cc:492] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:37.846618 15028 ts_tablet_manager.cc:1367] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:37.847005 15028 raft_consensus.cc:348] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.847097 15028 raft_consensus.cc:374] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:37.847124 15028 raft_consensus.cc:729] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:37.847224 15028 consensus_queue.cc:260] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.847486 15028 ts_tablet_manager.cc:1398] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:37.875875 8779 consensus_peers.cc:485] T 856c66722df446d1ae96fe248bc31941 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:37.932976 14844 raft_consensus.cc:482] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:37.933081 14844 raft_consensus.cc:504] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.933524 14844 leader_election.cc:288] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.933872 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:37.934000 6559 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.933982 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:37.934093 8575 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:37.934295 8910 leader_election.cc:302] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.934597 14844 raft_consensus.cc:2781] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:37.934655 14844 raft_consensus.cc:482] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:37.934705 14844 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.935873 14844 raft_consensus.cc:504] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.936244 14844 leader_election.cc:288] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:37.936479 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:37.936594 6559 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.936712 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91cdfb905944c1cbed56ab5cbd1253a" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:37.936863 8575 raft_consensus.cc:3037] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:37.937621 6559 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.937779 8575 raft_consensus.cc:2454] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:37.937958 8910 leader_election.cc:302] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:37.938097 14844 raft_consensus.cc:2781] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:37.938184 14844 raft_consensus.cc:686] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:37.938303 14844 consensus_queue.cc:237] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } }
I20240503 07:59:37.940865 6155 catalog_manager.cc:5484] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:38.020500 8779 consensus_peers.cc:485] T f269f05bd0bf4bf5b520f90e646b0171 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.032732 8779 consensus_peers.cc:485] T 63a31dc7bed443c196707da2988ce1a4 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.044904 8913 consensus_peers.cc:485] T d8c681e92f9748f2b6b20958a8efed36 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 07:59:38.052227 8913 consensus_peers.cc:485] T dbee3cf8da48461d94e1ad45ed06a88f P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:38.061272 6366 consensus_peers.cc:485] T 2a10085c34be4d31aae9f56a2ec1ba23 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.064613 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:38.068195 8575 raft_consensus.cc:1261] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.068303 6559 raft_consensus.cc:1261] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Refusing update from remote peer 3d08a06d53b14381bee6223c4e1a6027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.068573 15000 consensus_queue.cc:1035] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.068717 14844 consensus_queue.cc:1035] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.072466 6155 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.073763 6156 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.074162 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.074455 14992 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:38.074589 6150 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.076316 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.076738 15041 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.077096 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.077782 6150 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.078598 6148 catalog_manager.cc:5576] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949) does not have the latest schema for tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d]). Expected version 1 got 0
I20240503 07:59:38.076054 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.080302 15040 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.081465 6148 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 1)
I20240503 07:59:38.093695 14940 consensus_queue.cc:1035] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.096850 14940 consensus_queue.cc:1035] T 965f6a33bacb4776b59360d66a6d4cb1 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.123792 14928 consensus_queue.cc:1035] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.126530 14952 consensus_queue.cc:1035] T d74b95ba4bf64088898639dcd96f5730 P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.187203 6155 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 07:59:38.188987 8779 consensus_peers.cc:485] T 6317b68edd7c4512bb96248a9cac9399 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:38.193289 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.193423 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.193679 15041 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.193681 15040 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.193956 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.194221 14992 tablet.cc:1620] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
W20240503 07:59:38.198520 6366 consensus_peers.cc:485] T b61209b73f1f4872b5eccc823b1fa3df P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 07:59:38.200034 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 2)
I20240503 07:59:38.302707 6155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:38.302819 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:38.308914 15040 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.310362 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.310520 14992 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.310623 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.313084 15041 tablet.cc:1722] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.315804 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.315889 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 572903d1cc2544009f585ed18128b164 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.316049 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.316151 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 572903d1cc2544009f585ed18128b164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.318042 15028 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:38.318833 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.318943 15028 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.318994 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.319094 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.319260 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.320297 6670 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.320508 6670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.323681 15050 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.323737 6672 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb9abc63a05409bbc70f155dc4769fe (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323770 6541 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.323932 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb9abc63a05409bbc70f155dc4769fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324024 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.324251 15053 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:38.324554 8824 tablet_service.cc:1467] Processing CreateTablet for tablet be67ec23388948278f24987a5f6fa791 (DEFAULT_TABLE table=rebalancer_extra_table_72 [id=8cebff064c8e44a49e1aee352154d30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:38.324761 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be67ec23388948278f24987a5f6fa791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.325985 14984 consensus_queue.cc:1035] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.326215 15027 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:38.327090 15027 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.328081 6155 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d] alter complete (version 3)
I20240503 07:59:38.329108 15052 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:38.329409 15050 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330011 15053 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.330138 15052 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.331246 15049 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:38.332337 15050 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.332342 15028 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:38.332422 15028 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:38.332585 15049 tablet_bootstrap.cc:654] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.332593 15027 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:38.332661 15027 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:38.332677 15053 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:38.332749 15053 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.332917 15028 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.333030 15028 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.333063 15028 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.333139 15027 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.333220 15050 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:38.333179 15028 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.333241 15027 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.333268 15027 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.333364 15027 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.333444 15028 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.333588 15027 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.334805 15050 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.334877 15050 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.334898 15050 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.335000 15050 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.335265 15050 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 07:59:38.335436 15050 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.336148 15050 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.336679 15053 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.336802 15053 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.336835 15053 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.336936 15053 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.337203 15053 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:38.338912 15053 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:38.339661 15052 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:38.339756 15052 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 07:59:38.340034 15053 tablet_bootstrap.cc:654] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.340201 15052 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.340307 15052 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.340343 15052 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.340458 15052 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.340732 15052 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.340922 15053 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:38.340993 15053 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:38.341491 15053 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.341634 15053 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.341701 15053 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.341257 14928 consensus_queue.cc:1035] T 01384cbe92504a839474a60048fe293e P d1f36d57661b4429a4be8d10b58d8ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.341883 15053 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.342235 15053 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.342334 15049 tablet_bootstrap.cc:492] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:38.342433 15049 ts_tablet_manager.cc:1367] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 07:59:38.342864 15049 raft_consensus.cc:348] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.342972 15049 raft_consensus.cc:374] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.343016 15049 raft_consensus.cc:729] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.343163 15049 consensus_queue.cc:260] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.343492 15049 ts_tablet_manager.cc:1398] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:38.350200 15050 tablet_bootstrap.cc:492] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.350288 15050 ts_tablet_manager.cc:1367] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20240503 07:59:38.350850 15050 raft_consensus.cc:348] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.350962 15050 raft_consensus.cc:374] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.350997 15050 raft_consensus.cc:729] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.351109 15050 consensus_queue.cc:260] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.351358 15050 ts_tablet_manager.cc:1398] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.351548 15050 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Bootstrap starting.
I20240503 07:59:38.352948 15050 tablet_bootstrap.cc:654] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.359978 15050 tablet_bootstrap.cc:492] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: No bootstrap required, opened a new log
I20240503 07:59:38.360137 15050 ts_tablet_manager.cc:1367] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:38.360498 15050 raft_consensus.cc:348] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.360590 15050 raft_consensus.cc:374] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.360620 15050 raft_consensus.cc:729] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.360764 15050 consensus_queue.cc:260] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.361054 15050 ts_tablet_manager.cc:1398] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 07:59:38.375485 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.428201 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.428323 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:38.430336 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.430883 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:38.430883 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:55436
I20240503 07:59:38.430963 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet d91cdfb905944c1cbed56ab5cbd1253a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:38 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:38.431222 15066 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:38.431396 15066 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:38.431602 15066 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.432096 15066 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.433997 15066 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.434074 15066 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.434481 15066 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:38.435174 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.436924 15065 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:38.437047 15065 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.437152 15065 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.437376 15064 tablet_replica.cc:330] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: stopping tablet replica
I20240503 07:59:38.437511 15064 raft_consensus.cc:2229] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:38.437630 15064 raft_consensus.cc:2258] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:38.438071 15065 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.439730 15065 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.439801 15065 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.440054 15065 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:38.440620 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.440857 15064 ts_tablet_manager.cc:1875] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:38.442562 15064 ts_tablet_manager.cc:1888] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:38.442642 15064 log.cc:1196] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d91cdfb905944c1cbed56ab5cbd1253a
I20240503 07:59:38.442947 15064 ts_tablet_manager.cc:1909] T d91cdfb905944c1cbed56ab5cbd1253a P ad910f92ef9d4db8a8962cecba289f8a: Deleting consensus metadata
I20240503 07:59:38.443464 6141 catalog_manager.cc:4830] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693): tablet d91cdfb905944c1cbed56ab5cbd1253a (table rebalancer_dynamic_table_new_name [id=e5461e0a16bc46bbbae54f3f1df9d47d]) successfully deleted
I20240503 07:59:38.455526 15005 raft_consensus.cc:482] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.455636 15005 raft_consensus.cc:504] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.456001 15005 leader_election.cc:288] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.456339 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:38.456466 6429 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.456480 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:38.456602 8843 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.456888 6627 leader_election.cc:302] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, a97073f89fd44702bccb74a9b76f858d; no voters:
I20240503 07:59:38.457044 15005 raft_consensus.cc:2781] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.457108 15005 raft_consensus.cc:482] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.457134 15005 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.457949 15005 raft_consensus.cc:504] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.458245 15005 leader_election.cc:288] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.458498 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:38.458518 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb9abc63a05409bbc70f155dc4769fe" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:38.458593 8843 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.458603 6429 raft_consensus.cc:3037] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.459306 6429 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.459307 8843 raft_consensus.cc:2454] T 9eb9abc63a05409bbc70f155dc4769fe P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.459559 6628 leader_election.cc:302] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.459720 15005 raft_consensus.cc:2781] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.459798 15005 raft_consensus.cc:686] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.459951 15005 consensus_queue.cc:237] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.462224 6156 catalog_manager.cc:5484] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.474171 15005 raft_consensus.cc:482] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.474261 15005 raft_consensus.cc:504] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.474591 15005 leader_election.cc:288] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:38.474933 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:38.475024 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:38.475061 6298 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.475138 8575 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.475293 6625 leader_election.cc:302] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:38.475443 15005 raft_consensus.cc:2781] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.475497 15005 raft_consensus.cc:482] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.475530 15005 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.476256 15005 raft_consensus.cc:504] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.476505 15005 leader_election.cc:288] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267), c8d71787690245e681c85a087086e7ed (127.0.99.133:39949)
I20240503 07:59:38.476785 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:38.476836 8575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "572903d1cc2544009f585ed18128b164" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:38.476907 6298 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.476943 8575 raft_consensus.cc:3037] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.477825 8575 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.477825 6298 raft_consensus.cc:2454] T 572903d1cc2544009f585ed18128b164 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.478091 6625 leader_election.cc:302] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:38.478222 15005 raft_consensus.cc:2781] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.478297 15005 raft_consensus.cc:686] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.478417 15005 consensus_queue.cc:237] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } }
I20240503 07:59:38.480901 6156 catalog_manager.cc:5484] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:38.531988 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:38.537221 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.537262 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.537465 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.537473 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.539023 15028 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:38.539320 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]), partition=
I20240503 07:59:38.539549 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa21f731dfe4a1a938ca51fe59989fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:38.539925 15028 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.540753 15069 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:38.540964 15028 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:38.541041 15028 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:38.541451 15028 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.541535 15028 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.541563 15028 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.541669 15028 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.541700 15069 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.541893 15052 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:38.541910 15028 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:38.542721 15052 tablet_bootstrap.cc:654] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:38.543191 15069 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:38.543277 15069 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.543956 15069 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.544041 15069 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.544070 15069 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.544203 15069 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.544718 15052 tablet_bootstrap.cc:492] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:38.544831 15052 ts_tablet_manager.cc:1367] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:38.545254 15052 raft_consensus.cc:348] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.545342 15052 raft_consensus.cc:374] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:38.545379 15052 raft_consensus.cc:729] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:38.545486 15069 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:38.545496 15052 consensus_queue.cc:260] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.545765 15052 ts_tablet_manager.cc:1398] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:38.557619 14427 raft_consensus.cc:482] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.557714 14427 raft_consensus.cc:504] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.558079 14427 leader_election.cc:288] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:38.558432 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2" is_pre_election: true
I20240503 07:59:38.558554 8843 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.558725 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:38.558844 6559 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 0.
I20240503 07:59:38.559124 6625 leader_election.cc:302] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:38.559373 14427 raft_consensus.cc:2781] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.559435 14427 raft_consensus.cc:482] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.559464 14427 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.560062 15034 raft_consensus.cc:482] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:38.560132 15034 raft_consensus.cc:504] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.560431 15034 leader_election.cc:288] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.560407 14427 raft_consensus.cc:504] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.560724 14427 leader_election.cc:288] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Requested vote from peers 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135:39417), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:38.560731 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:38.560881 6429 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:38.560875 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:38.560967 8976 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 0.
I20240503 07:59:38.561094 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:38.561168 8517 leader_election.cc:302] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.561193 6559 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.561319 15034 raft_consensus.cc:2781] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:38.561362 15034 raft_consensus.cc:482] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:38.561390 15034 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.561949 6559 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.562172 15034 raft_consensus.cc:504] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.562201 6625 leader_election.cc:302] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97073f89fd44702bccb74a9b76f858d, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:38.562568 14427 raft_consensus.cc:2781] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.562614 15034 leader_election.cc:288] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:38.562590 8843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be67ec23388948278f24987a5f6fa791" candidate_uuid: "a97073f89fd44702bccb74a9b76f858d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616fdda6ba714cb7be01a564de2b9db2"
I20240503 07:59:38.562690 8843 raft_consensus.cc:3037] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.562736 14427 raft_consensus.cc:686] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [term 1 LEADER]: Becoming Leader. State: Replica: a97073f89fd44702bccb74a9b76f858d, State: Running, Role: LEADER
I20240503 07:59:38.562858 14427 consensus_queue.cc:237] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:38.562922 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:38.563002 6429 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.563599 8843 raft_consensus.cc:2454] T be67ec23388948278f24987a5f6fa791 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97073f89fd44702bccb74a9b76f858d in term 1.
I20240503 07:59:38.563791 6429 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:38.564133 8517 leader_election.cc:302] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d71787690245e681c85a087086e7ed, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:38.564397 15034 raft_consensus.cc:2781] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:38.564492 15034 raft_consensus.cc:686] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Becoming Leader. State: Replica: c8d71787690245e681c85a087086e7ed, State: Running, Role: LEADER
I20240503 07:59:38.564611 15034 consensus_queue.cc:237] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:38.564667 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa21f731dfe4a1a938ca51fe59989fe" candidate_uuid: "c8d71787690245e681c85a087086e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:38.564759 8976 raft_consensus.cc:3037] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:38.565639 8976 raft_consensus.cc:2454] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d71787690245e681c85a087086e7ed in term 1.
I20240503 07:59:38.565661 6150 catalog_manager.cc:5484] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d reported cstate change: term changed from 0 to 1, leader changed from <none> to a97073f89fd44702bccb74a9b76f858d (127.0.99.132). New cstate: current_term: 1 leader_uuid: "a97073f89fd44702bccb74a9b76f858d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a97073f89fd44702bccb74a9b76f858d" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 39733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:38.567929 6148 catalog_manager.cc:5484] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d71787690245e681c85a087086e7ed (127.0.99.133). New cstate: current_term: 1 leader_uuid: "c8d71787690245e681c85a087086e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:38.640483 8913 consensus_peers.cc:485] T 2c6db695ea2a48c49e687d95df2af58d P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.674605 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:38.678755 6429 raft_consensus.cc:1261] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.679095 15034 consensus_queue.cc:1035] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.679170 8976 raft_consensus.cc:1261] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer c8d71787690245e681c85a087086e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:38.679497 15034 consensus_queue.cc:1035] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.682077 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.682380 15041 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.682611 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]). Expected version 1 got 0
I20240503 07:59:38.683173 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.683243 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:38.683444 15080 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:38.683521 15040 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.684135 6148 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4]). Expected version 1 got 0
I20240503 07:59:38.689302 6156 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 1)
W20240503 07:59:38.755965 6144 catalog_manager.cc:4573] TS 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d78407a3354848826ab84e5b750ce0: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.791808 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 07:59:38.797223 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.797554 15041 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: MemRowSet was empty: no flush needed.
I20240503 07:59:38.797706 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.798022 15040 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:38.801735 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 07:59:38.802012 15080 tablet.cc:1620] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:38.802950 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 2)
W20240503 07:59:38.888700 6366 consensus_peers.cc:485] T edf2c780d0d64db79dca88eac77d4478 P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:38.902137 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
I20240503 07:59:38.906383 6150 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 07:59:38.911391 15041 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.911795 15040 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.913933 15080 tablet.cc:1722] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 07:59:38.916520 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4] alter complete (version 3)
W20240503 07:59:38.917145 6235 consensus_peers.cc:485] T c8cf3d6a3ace46769854ed61f340fb25 P c167a0a433d34b74affed13cbf96e302 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:38.944195 15073 consensus_queue.cc:1035] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.944716 15073 consensus_queue.cc:1035] T 572903d1cc2544009f585ed18128b164 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.979795 15073 consensus_queue.cc:1035] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.983799 15073 consensus_queue.cc:1035] T 9eb9abc63a05409bbc70f155dc4769fe P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.984558 15073 consensus_queue.cc:1035] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:38.988646 14427 consensus_queue.cc:1035] T be67ec23388948278f24987a5f6fa791 P a97073f89fd44702bccb74a9b76f858d [LEADER]: Connected to new peer: Peer: permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.018988 6156 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.019140 6156 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 07:59:39.021430 6156 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: Sending DeleteTablet for 3 replicas of tablet 6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.021982 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:52420
I20240503 07:59:39.022006 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:45776
I20240503 07:59:39.022002 6408 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa21f731dfe4a1a938ca51fe59989fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 07:59:39 UTC) from {username='slave'} at 127.0.0.1:44410
I20240503 07:59:39.022418 15093 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: stopping tablet replica
I20240503 07:59:39.022470 15095 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: stopping tablet replica
I20240503 07:59:39.022631 15094 tablet_replica.cc:330] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: stopping tablet replica
I20240503 07:59:39.022647 15095 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.022758 15094 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 07:59:39.022763 15095 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.023069 15093 raft_consensus.cc:2229] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 LEADER]: Raft consensus shutting down.
I20240503 07:59:39.023324 15093 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.023361 15095 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.023438 15094 raft_consensus.cc:2258] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 07:59:39.024219 15094 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.024940 15093 ts_tablet_manager.cc:1875] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 07:59:39.026530 15095 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.026616 15095 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.026937 15095 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P d1f36d57661b4429a4be8d10b58d8ff7: Deleting consensus metadata
I20240503 07:59:39.027565 6144 catalog_manager.cc:4830] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
I20240503 07:59:39.029206 15094 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.029281 15094 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.029693 15093 ts_tablet_manager.cc:1888] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 07:59:39.029763 15093 log.cc:1196] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting WAL directory at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/6fa21f731dfe4a1a938ca51fe59989fe
I20240503 07:59:39.029980 15093 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P c8d71787690245e681c85a087086e7ed: Deleting consensus metadata
I20240503 07:59:39.030470 6141 catalog_manager.cc:4830] TS c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
I20240503 07:59:39.030573 15094 ts_tablet_manager.cc:1909] T 6fa21f731dfe4a1a938ca51fe59989fe P 3d08a06d53b14381bee6223c4e1a6027: Deleting consensus metadata
I20240503 07:59:39.031148 6143 catalog_manager.cc:4830] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089): tablet 6fa21f731dfe4a1a938ca51fe59989fe (table rebalancer_dynamic_table_new_name [id=c068f2b7a74f480c8921566676834cf4]) successfully deleted
W20240503 07:59:39.036003 6628 consensus_peers.cc:485] T 2054e7790d7745c2a1b08e903c64d1b7 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.044960 8517 consensus_peers.cc:485] T f534eefb7c334889a9c5d02175ae2d94 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.080870 8517 consensus_peers.cc:485] T ef76600ded194f318cf97f157845bea8 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.113070 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_extra_table_74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:39.119096 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.119383 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.119410 6278 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.119712 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.120203 8824 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.120458 8824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.121683 15098 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:39.122431 8560 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.122651 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.122651 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 25d97fed055741a8a9eb6696fd791a5b (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.122792 15098 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.122897 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d97fed055741a8a9eb6696fd791a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.123910 6156 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57060:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 07:59:39.123992 15098 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:39.124080 15098 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:39.124524 15098 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.124627 15098 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.124660 15098 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.124867 15098 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.125023 8955 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.125274 8955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.125865 15098 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.127149 15099 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Bootstrap starting.
I20240503 07:59:39.127827 15101 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.127900 6278 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.128125 6278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.128636 6541 tablet_service.cc:1467] Processing CreateTablet for tablet b4f99fb1e97245bcafbaceaeb34f5d70 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.128990 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f99fb1e97245bcafbaceaeb34f5d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.129673 15097 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:39.130306 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 6511469dd3b84dd7802833602176fb71 (DEFAULT_TABLE table=rebalancer_extra_table_74 [id=ac8b71b05a3d46feac2d52613d80473b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:39.130518 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6511469dd3b84dd7802833602176fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.130704 15099 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.130738 15097 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.132953 15098 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Bootstrap starting.
I20240503 07:59:39.132288 15097 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:39.133427 15097 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:39.133848 15098 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.133946 15097 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.134050 15097 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.134073 15097 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.134164 15097 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.133064 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.134533 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.134981 6408 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.135206 6408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.135640 15097 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.135782 15097 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Bootstrap starting.
I20240503 07:59:39.136718 15097 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.138623 15101 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.138818 15106 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Bootstrap starting.
I20240503 07:59:39.139364 15102 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.139814 15099 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: No bootstrap required, opened a new log
I20240503 07:59:39.140129 15099 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 07:59:39.140276 6539 tablet_service.cc:1467] Processing CreateTablet for tablet 0052e30f9af04cce875ace9d064c3896 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]), partition=
I20240503 07:59:39.140393 15102 tablet_bootstrap.cc:654] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.140465 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0052e30f9af04cce875ace9d064c3896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.140578 15099 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.140672 15099 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.140705 15099 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.140831 15099 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.141083 15106 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.141541 15101 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.141669 15101 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 07:59:39.142156 15101 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.142235 15099 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:39.142297 15101 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.142370 15101 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.142514 15101 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.142848 15101 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.142974 15097 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: No bootstrap required, opened a new log
I20240503 07:59:39.143055 15097 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 07:59:39.143399 15097 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.143484 15097 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.143513 15097 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d71787690245e681c85a087086e7ed, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.143635 15097 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.143872 15097 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.143985 15101 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.144912 15101 tablet_bootstrap.cc:654] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.146865 15098 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: No bootstrap required, opened a new log
I20240503 07:59:39.146952 15098 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 07:59:39.147449 15098 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.147548 15098 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.147581 15098 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.147593 15106 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: No bootstrap required, opened a new log
I20240503 07:59:39.147656 15106 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 07:59:39.147696 15098 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.147974 15098 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.148046 15106 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.148136 15106 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.148165 15106 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f36d57661b4429a4be8d10b58d8ff7, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.148294 15106 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.150408 15106 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:39.151146 15101 tablet_bootstrap.cc:492] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.151247 15101 ts_tablet_manager.cc:1367] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 07:59:39.151639 15101 raft_consensus.cc:348] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.151752 15101 raft_consensus.cc:374] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.151798 15101 raft_consensus.cc:729] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.151921 15101 consensus_queue.cc:260] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.152915 15101 ts_tablet_manager.cc:1398] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:39.153110 15101 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Bootstrap starting.
I20240503 07:59:39.154011 15101 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.161590 15102 tablet_bootstrap.cc:492] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.161681 15102 ts_tablet_manager.cc:1367] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 07:59:39.162042 15102 raft_consensus.cc:348] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.162132 15102 raft_consensus.cc:374] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.162164 15102 raft_consensus.cc:729] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.162290 15102 consensus_queue.cc:260] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.162532 15102 ts_tablet_manager.cc:1398] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:39.162716 15102 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.164064 15102 tablet_bootstrap.cc:654] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.169732 14940 raft_consensus.cc:482] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.169806 14940 raft_consensus.cc:504] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.170131 14940 leader_election.cc:288] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.170454 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:39.170471 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:39.170552 6559 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:39.170583 8581 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 0.
I20240503 07:59:39.170985 6232 leader_election.cc:302] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c167a0a433d34b74affed13cbf96e302, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:39.171229 14940 raft_consensus.cc:2781] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.171290 14940 raft_consensus.cc:482] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.171368 14940 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.171770 15102 tablet_bootstrap.cc:492] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.171849 15102 ts_tablet_manager.cc:1367] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 07:59:39.172252 15102 raft_consensus.cc:348] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.172369 15102 raft_consensus.cc:374] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.172403 15102 raft_consensus.cc:729] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.172528 15102 consensus_queue.cc:260] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.172650 14940 raft_consensus.cc:504] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.172848 15102 ts_tablet_manager.cc:1398] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.173015 15102 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Bootstrap starting.
I20240503 07:59:39.173041 14940 leader_election.cc:288] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.174439 15102 tablet_bootstrap.cc:654] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.175134 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:39.175241 6559 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.175230 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f99fb1e97245bcafbaceaeb34f5d70" candidate_uuid: "c167a0a433d34b74affed13cbf96e302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:39.175336 8581 raft_consensus.cc:3037] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.176095 8581 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:39.176103 6559 raft_consensus.cc:2454] T b4f99fb1e97245bcafbaceaeb34f5d70 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c167a0a433d34b74affed13cbf96e302 in term 1.
I20240503 07:59:39.176510 6232 leader_election.cc:302] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:39.176970 14985 raft_consensus.cc:2781] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.177127 14985 raft_consensus.cc:686] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [term 1 LEADER]: Becoming Leader. State: Replica: c167a0a433d34b74affed13cbf96e302, State: Running, Role: LEADER
I20240503 07:59:39.177251 14985 consensus_queue.cc:237] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } }
I20240503 07:59:39.178089 15101 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: No bootstrap required, opened a new log
I20240503 07:59:39.178174 15101 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.001s
I20240503 07:59:39.178562 15101 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.178746 15101 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.178824 15101 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.179004 15101 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.180197 15101 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 07:59:39.183171 15102 tablet_bootstrap.cc:492] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: No bootstrap required, opened a new log
I20240503 07:59:39.183264 15102 ts_tablet_manager.cc:1367] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 07:59:39.183611 6150 catalog_manager.cc:5484] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 reported cstate change: term changed from 0 to 1, leader changed from <none> to c167a0a433d34b74affed13cbf96e302 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c167a0a433d34b74affed13cbf96e302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.183697 15102 raft_consensus.cc:348] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.183791 15102 raft_consensus.cc:374] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.183833 15102 raft_consensus.cc:729] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.183948 15102 consensus_queue.cc:260] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.184214 15102 ts_tablet_manager.cc:1398] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:39.201299 14944 raft_consensus.cc:482] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.201419 14944 raft_consensus.cc:504] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.201783 14944 leader_election.cc:288] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:39.202091 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302" is_pre_election: true
I20240503 07:59:39.202209 6298 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:39.202247 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed" is_pre_election: true
I20240503 07:59:39.202368 8581 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 0.
I20240503 07:59:39.202559 8910 leader_election.cc:302] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c167a0a433d34b74affed13cbf96e302; no voters:
I20240503 07:59:39.202714 14944 raft_consensus.cc:2781] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.202772 14944 raft_consensus.cc:482] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.202809 14944 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.203603 14944 raft_consensus.cc:504] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.203858 14944 leader_election.cc:288] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Requested vote from peers c8d71787690245e681c85a087086e7ed (127.0.99.133:39949), c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267)
I20240503 07:59:39.204126 8581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d71787690245e681c85a087086e7ed"
I20240503 07:59:39.204155 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d97fed055741a8a9eb6696fd791a5b" candidate_uuid: "3d08a06d53b14381bee6223c4e1a6027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c167a0a433d34b74affed13cbf96e302"
I20240503 07:59:39.204237 6298 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.204237 8581 raft_consensus.cc:3037] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.205155 8581 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c8d71787690245e681c85a087086e7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:39.205155 6298 raft_consensus.cc:2454] T 25d97fed055741a8a9eb6696fd791a5b P c167a0a433d34b74affed13cbf96e302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d08a06d53b14381bee6223c4e1a6027 in term 1.
I20240503 07:59:39.205422 8910 leader_election.cc:302] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, c8d71787690245e681c85a087086e7ed; no voters:
I20240503 07:59:39.205588 14944 raft_consensus.cc:2781] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.205672 14944 raft_consensus.cc:686] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [term 1 LEADER]: Becoming Leader. State: Replica: 3d08a06d53b14381bee6223c4e1a6027, State: Running, Role: LEADER
I20240503 07:59:39.205785 14944 consensus_queue.cc:237] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } }
I20240503 07:59:39.207787 6148 catalog_manager.cc:5484] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "3d08a06d53b14381bee6223c4e1a6027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c167a0a433d34b74affed13cbf96e302" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:39.241195 6628 consensus_peers.cc:485] T 435f73fb9f384b10b353df19037bcac3 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.247099 6497 consensus_peers.cc:485] T afcd161598db41d7962caca61f63ef77 P ad910f92ef9d4db8a8962cecba289f8a -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.286736 6366 consensus_peers.cc:462] T 60f643dce3eb4c669b36b0714a31651b P d1f36d57661b4429a4be8d10b58d8ff7 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111)
W20240503 07:59:39.297787 8517 consensus_peers.cc:485] T 00bcbd5440ff4d56b1975edc4041b428 P c8d71787690245e681c85a087086e7ed -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.300087 15004 raft_consensus.cc:482] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.300184 15004 raft_consensus.cc:504] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.300563 15004 leader_election.cc:288] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.300921 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a" is_pre_election: true
I20240503 07:59:39.300945 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:39.301048 6559 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:39.301054 8976 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 0.
I20240503 07:59:39.301307 8778 leader_election.cc:302] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, 616fdda6ba714cb7be01a564de2b9db2; no voters:
I20240503 07:59:39.301497 14955 raft_consensus.cc:2781] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.301551 14955 raft_consensus.cc:482] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.301589 14955 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.302649 14955 raft_consensus.cc:504] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.302932 14955 leader_election.cc:288] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693)
I20240503 07:59:39.303194 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad910f92ef9d4db8a8962cecba289f8a"
I20240503 07:59:39.303282 6559 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.303195 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6511469dd3b84dd7802833602176fb71" candidate_uuid: "616fdda6ba714cb7be01a564de2b9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:39.303355 8976 raft_consensus.cc:3037] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.304177 6559 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:39.304467 8776 leader_election.cc:302] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616fdda6ba714cb7be01a564de2b9db2, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:39.304482 8976 raft_consensus.cc:2454] T 6511469dd3b84dd7802833602176fb71 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 616fdda6ba714cb7be01a564de2b9db2 in term 1.
I20240503 07:59:39.304618 14955 raft_consensus.cc:2781] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.304719 14955 raft_consensus.cc:686] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 616fdda6ba714cb7be01a564de2b9db2, State: Running, Role: LEADER
I20240503 07:59:39.304885 14955 consensus_queue.cc:237] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } }
I20240503 07:59:39.307137 6148 catalog_manager.cc:5484] T 6511469dd3b84dd7802833602176fb71 P 616fdda6ba714cb7be01a564de2b9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 616fdda6ba714cb7be01a564de2b9db2 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "616fdda6ba714cb7be01a564de2b9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "616fdda6ba714cb7be01a564de2b9db2" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 39417 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:39.309943 8913 consensus_peers.cc:485] T a833fb4980c04673a335d020a3ad3dd3 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 07:59:39.315850 6628 consensus_peers.cc:485] T 9772b487ca864ad98e9d79d8435c34c6 P a97073f89fd44702bccb74a9b76f858d -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 07:59:39.379700 8913 consensus_peers.cc:485] T 873c7786052b4b9c97cc0c1bcb6e7a03 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer 9553fe3c4ccd4e91b7c176aad82fec09 (127.0.99.134:40925): Couldn't send request to peer 9553fe3c4ccd4e91b7c176aad82fec09. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:40925: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 07:59:39.415431 15035 raft_consensus.cc:482] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:39.415555 15035 raft_consensus.cc:504] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.415939 15035 leader_election.cc:288] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:39.416292 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" is_pre_election: true
I20240503 07:59:39.416283 8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027" is_pre_election: true
I20240503 07:59:39.416428 6429 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:39.416455 8976 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 0.
I20240503 07:59:39.416739 6496 leader_election.cc:302] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d08a06d53b14381bee6223c4e1a6027, ad910f92ef9d4db8a8962cecba289f8a; no voters:
I20240503 07:59:39.417025 15035 raft_consensus.cc:2781] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:39.417136 15035 raft_consensus.cc:482] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:39.417237 15035 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.418284 15035 raft_consensus.cc:504] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.418665 15035 leader_election.cc:288] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Requested vote from peers 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089), d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651)
I20240503 07:59:39.419080 8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d08a06d53b14381bee6223c4e1a6027"
I20240503 07:59:39.419129 6429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0052e30f9af04cce875ace9d064c3896" candidate_uuid: "ad910f92ef9d4db8a8962cecba289f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f36d57661b4429a4be8d10b58d8ff7"
I20240503 07:59:39.419188 8975 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.419227 6429 raft_consensus.cc:3037] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.419957 6429 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:39.420090 8975 raft_consensus.cc:2454] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad910f92ef9d4db8a8962cecba289f8a in term 1.
I20240503 07:59:39.420257 6497 leader_election.cc:302] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad910f92ef9d4db8a8962cecba289f8a, d1f36d57661b4429a4be8d10b58d8ff7; no voters:
I20240503 07:59:39.420461 15035 raft_consensus.cc:2781] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.420596 15035 raft_consensus.cc:686] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [term 1 LEADER]: Becoming Leader. State: Replica: ad910f92ef9d4db8a8962cecba289f8a, State: Running, Role: LEADER
I20240503 07:59:39.420729 15035 consensus_queue.cc:237] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } }
I20240503 07:59:39.423180 6148 catalog_manager.cc:5484] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "ad910f92ef9d4db8a8962cecba289f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:39.538694 6148 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57060:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 07:59:39.543291 8975 raft_consensus.cc:1261] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.543279 6429 raft_consensus.cc:1261] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7 [term 1 FOLLOWER]: Refusing update from remote peer ad910f92ef9d4db8a8962cecba289f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:39.543730 15035 consensus_queue.cc:1035] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f36d57661b4429a4be8d10b58d8ff7" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.543968 15096 consensus_queue.cc:1035] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d08a06d53b14381bee6223c4e1a6027" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 41089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.546836 6148 catalog_manager.cc:5576] TS d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.547971 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.548426 15119 mvcc.cc:205] Tried to move back new op lower bound from 7023506143406202880 to 7023506142908743680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506143406202880}]
I20240503 07:59:39.548573 15122 mvcc.cc:205] Tried to move back new op lower bound from 7023506143406202880 to 7023506142908743680. Current Snapshot: MvccSnapshot[applied={T|T < 7023506143406202880}]
I20240503 07:59:39.549748 15124 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.549880 6156 catalog_manager.cc:5576] TS ad910f92ef9d4db8a8962cecba289f8a (127.0.99.131:45693) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.550029 15125 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.550220 15124 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P ad910f92ef9d4db8a8962cecba289f8a: MemRowSet was empty: no flush needed.
I20240503 07:59:39.550570 15126 tablet.cc:1722] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 07:59:39.550966 15126 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P d1f36d57661b4429a4be8d10b58d8ff7: MemRowSet was empty: no flush needed.
I20240503 07:59:39.552459 6156 catalog_manager.cc:5576] TS 3d08a06d53b14381bee6223c4e1a6027 (127.0.99.136:41089) does not have the latest schema for tablet 0052e30f9af04cce875ace9d064c3896 (table rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3]). Expected version 1 got 0
I20240503 07:59:39.554541 6150 catalog_manager.cc:6029] T 00000000000000000000000000000000 P e0b9972b770f4d3bb0f76d2fbe669505: rebalancer_dynamic_table [id=4fe18d99b08d41cbafee35c9aec886d3] alter complete (version 1)
I20240503 07:59:39.554973 15125 tablet.cc:1620] T 0052e30f9af04cce875ace9d064c3896 P 3d08a06d53b14381bee6223c4e1a6027: MemRowSet was empty: no flush needed.
I20240503 07:59:39.585461 15076 consensus_queue.cc:1035] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.588193 14940 consensus_queue.cc:1035] T b4f99fb1e97245bcafbaceaeb34f5d70 P c167a0a433d34b74affed13cbf96e302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad910f92ef9d4db8a8962cecba289f8a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:39.603827 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6217
I20240503 07:59:39.608659 14944 consensus_queue.cc:1035] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d71787690245e681c85a087086e7ed" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:39.613623 8910 connection.cc:535] client connection to 127.0.99.129:37267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:39.614406 8910 consensus_peers.cc:485] T 25d97fed055741a8a9eb6696fd791a5b P 3d08a06d53b14381bee6223c4e1a6027 -> Peer c167a0a433d34b74affed13cbf96e302 (127.0.99.129:37267): Couldn't send request to peer c167a0a433d34b74affed13cbf96e302. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:37267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.614866 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6348
I20240503 07:59:39.626412 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6479
W20240503 07:59:39.626700 13845 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:39.625819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.130:41651 (local address 127.0.99.135:60133)
0503 07:59:39.625963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:39.626005 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":74}
W20240503 07:59:39.627806 8779 consensus_peers.cc:485] T fc7d26466129405bbe6db2721823bae1 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.635089 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6610
I20240503 07:59:39.642609 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8499
W20240503 07:59:39.649338 8776 connection.cc:535] client connection to 127.0.99.133:39949 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 07:59:39.649621 8779 consensus_peers.cc:485] T 5b745ba777b2476fbae78706b35125f6 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer d1f36d57661b4429a4be8d10b58d8ff7 (127.0.99.130:41651): Couldn't send request to peer d1f36d57661b4429a4be8d10b58d8ff7. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.130:41651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 07:59:39.650115 15132 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 07:59:39.649758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:39949 (local address 127.0.99.135:41463)
0503 07:59:39.649853 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0503 07:59:39.649864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0503 07:59:39.649956 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 07:59:39.650032 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:39949: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
W20240503 07:59:39.650213 8776 consensus_peers.cc:485] T 253d36eba03243c39dfbb4ca750383c9 P 616fdda6ba714cb7be01a564de2b9db2 -> Peer c8d71787690245e681c85a087086e7ed (127.0.99.133:39949): Couldn't send request to peer c8d71787690245e681c85a087086e7ed. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:39949: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.650342 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8761
W20240503 07:59:39.657717 8912 consensus_peers.cc:485] T 6246801a888a4ff3b280b812de0318e1 P 3d08a06d53b14381bee6223c4e1a6027 -> Peer a97073f89fd44702bccb74a9b76f858d (127.0.99.132:39733): Couldn't send request to peer a97073f89fd44702bccb74a9b76f858d. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.132:39733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 07:59:39.658046 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 8894
I20240503 07:59:39.664227 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 6126
2024-05-03T07:59:39Z chronyd exiting
I20240503 07:59:39.680066 398 test_util.cc:175] -----------------------------------------------
I20240503 07:59:39.680104 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714723112017232-398-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (46236 ms)
[----------] 1 test from DDLDuringRebalancingTest (46236 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240503 07:59:39.680720 398 test_util.cc:268] Using random seed: 436425678
I20240503 07:59:39.681521 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 07:59:39.681553 398 ts_itest-base.cc:113] --------------
I20240503 07:59:39.681568 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 07:59:39.681581 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 07:59:39.681594 398 ts_itest-base.cc:116] --------------
2024-05-03T07:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T07:59:39Z Disabled control of system clock
I20240503 07:59:39.688357 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43357
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:43357
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 07:59:39.786161 15140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:39.786365 15140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:39.786425 15140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:39.789292 15140 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:39.789363 15140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 07:59:39.789400 15140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:39.789438 15140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 07:59:39.789474 15140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 07:59:39.792928 15140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:43357
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:39.793653 15140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:39.794644 15140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:39.801930 15146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:39.802649 15145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:39.803189 15148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:39.804268 15140 server_base.cc:1034] running on GCE node
I20240503 07:59:39.804692 15140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:39.805528 15140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:39.806668 15140 hybrid_clock.cc:648] HybridClock initialized: now 1714723179806640 us; error 40 us; skew 500 ppm
I20240503 07:59:39.813588 15140 webserver.cc:460] Webserver started at http://127.0.99.190:35835/ using document root <none> and password file <none>
I20240503 07:59:39.814069 15140 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:39.814119 15140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:39.814271 15140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:39.815774 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "57b53b572dff47a5b3c9c045f82c3b57"
format_stamp: "Formatted at 2024-05-03 07:59:39 on dist-test-slave-7tt0"
I20240503 07:59:39.816202 15140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "57b53b572dff47a5b3c9c045f82c3b57"
format_stamp: "Formatted at 2024-05-03 07:59:39 on dist-test-slave-7tt0"
I20240503 07:59:39.819180 15140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:39.821254 15154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:39.822271 15140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:39.822381 15140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "57b53b572dff47a5b3c9c045f82c3b57"
format_stamp: "Formatted at 2024-05-03 07:59:39 on dist-test-slave-7tt0"
I20240503 07:59:39.822460 15140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:39.834913 15140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:39.835412 15140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:39.835546 15140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:39.841586 15140 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:43357
I20240503 07:59:39.841594 15206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:43357 every 8 connection(s)
I20240503 07:59:39.842458 15140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 07:59:39.844753 15207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:39.849279 15207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57: Bootstrap starting.
I20240503 07:59:39.851239 15207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:39.851835 15207 log.cc:824] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:39.852532 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15140
I20240503 07:59:39.852663 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 07:59:39.853544 15207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57: No bootstrap required, opened a new log
I20240503 07:59:39.856487 15207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } }
I20240503 07:59:39.856658 15207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:39.856734 15207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b53b572dff47a5b3c9c045f82c3b57, State: Initialized, Role: FOLLOWER
I20240503 07:59:39.857129 15207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } }
I20240503 07:59:39.857249 15207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 07:59:39.857311 15207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 07:59:39.857421 15207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:39.858076 15207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } }
I20240503 07:59:39.858404 15207 leader_election.cc:302] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b53b572dff47a5b3c9c045f82c3b57; no voters:
I20240503 07:59:39.858690 15207 leader_election.cc:288] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 07:59:39.858778 15212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:39.858947 15212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [term 1 LEADER]: Becoming Leader. State: Replica: 57b53b572dff47a5b3c9c045f82c3b57, State: Running, Role: LEADER
I20240503 07:59:39.859211 15212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } }
I20240503 07:59:39.859721 15207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 07:59:39.860374 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b53b572dff47a5b3c9c045f82c3b57. Latest consensus state: current_term: 1 leader_uuid: "57b53b572dff47a5b3c9c045f82c3b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } } }
I20240503 07:59:39.860484 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:39.860613 15213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b53b572dff47a5b3c9c045f82c3b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b53b572dff47a5b3c9c045f82c3b57" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 43357 } } }
I20240503 07:59:39.860708 15213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57 [sys.catalog]: This master's current role is: LEADER
I20240503 07:59:39.860857 15216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 07:59:39.863698 15216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 07:59:39.868239 15216 catalog_manager.cc:1347] Generated new cluster ID: 5851aab5dc1a4aaf9c6be511903a83c1
I20240503 07:59:39.868310 15216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 07:59:39.886327 15216 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 07:59:39.887387 15216 catalog_manager.cc:1504] Loading token signing keys...
I20240503 07:59:39.907235 15216 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 57b53b572dff47a5b3c9c045f82c3b57: Generated new TSK 0
I20240503 07:59:39.907889 15216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 07:59:39.912274 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:40.010604 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.010879 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.010974 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.013940 15231 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.013995 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.014075 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 07:59:40.017480 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.018200 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.019153 15231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.027037 15239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.027568 15236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.028471 15237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.028995 15231 server_base.cc:1034] running on GCE node
I20240503 07:59:40.029345 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.029867 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.031023 15231 hybrid_clock.cc:648] HybridClock initialized: now 1714723180030996 us; error 39 us; skew 500 ppm
I20240503 07:59:40.032708 15231 webserver.cc:460] Webserver started at http://127.0.99.129:43453/ using document root <none> and password file <none>
I20240503 07:59:40.033216 15231 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.033273 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.033440 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.034962 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c770157b083143399bcf339b4081a23f"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.035416 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c770157b083143399bcf339b4081a23f"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.038534 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:40.040473 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.041359 15231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:40.041460 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c770157b083143399bcf339b4081a23f"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.041539 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.068857 15231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.069401 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.069557 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.070047 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.070900 15231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.070951 15231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.071005 15231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.071036 15231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.079392 15231 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:36571
I20240503 07:59:40.079412 15358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:36571 every 8 connection(s)
I20240503 07:59:40.080256 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 07:59:40.087311 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15231
I20240503 07:59:40.087414 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 07:59:40.088546 15359 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.088760 15359 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.089255 15359 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.089461 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:40.090531 15170 ts_manager.cc:194] Registered new tserver with Master: c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:40.091609 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:55227
W20240503 07:59:40.187366 15362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.187546 15362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.187608 15362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.190485 15362 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.190543 15362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.190630 15362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 07:59:40.193957 15362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.194655 15362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.195598 15362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.203539 15370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.204294 15367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.204793 15368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.205452 15362 server_base.cc:1034] running on GCE node
I20240503 07:59:40.205751 15362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.206182 15362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.207315 15362 hybrid_clock.cc:648] HybridClock initialized: now 1714723180207293 us; error 39 us; skew 500 ppm
I20240503 07:59:40.208840 15362 webserver.cc:460] Webserver started at http://127.0.99.130:42905/ using document root <none> and password file <none>
I20240503 07:59:40.209306 15362 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.209359 15362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.209513 15362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.210917 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "07ae48a2fef84a10a186e6256e28c064"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.211315 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "07ae48a2fef84a10a186e6256e28c064"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.214186 15362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:40.216050 15376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.216921 15362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:40.217010 15362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "07ae48a2fef84a10a186e6256e28c064"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.217082 15362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.228737 15362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.229207 15362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.229333 15362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.229743 15362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.230460 15362 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.230509 15362 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.230554 15362 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.230579 15362 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.238134 15362 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:36961
I20240503 07:59:40.238173 15489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:36961 every 8 connection(s)
I20240503 07:59:40.238890 15362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 07:59:40.244987 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15362
I20240503 07:59:40.245123 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 07:59:40.246840 15490 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.247017 15490 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.247136 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:40.247483 15490 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.248382 15170 ts_manager.cc:194] Registered new tserver with Master: 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:40.248982 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:52371
W20240503 07:59:40.348044 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.348215 15493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.348271 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.351161 15493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.351222 15493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.351297 15493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 07:59:40.354794 15493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.355551 15493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.356528 15493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.364717 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.365406 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.365917 15501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.366632 15493 server_base.cc:1034] running on GCE node
I20240503 07:59:40.366961 15493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.367446 15493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.368602 15493 hybrid_clock.cc:648] HybridClock initialized: now 1714723180368592 us; error 25 us; skew 500 ppm
I20240503 07:59:40.370270 15493 webserver.cc:460] Webserver started at http://127.0.99.131:35111/ using document root <none> and password file <none>
I20240503 07:59:40.370764 15493 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.370817 15493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.370972 15493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.372447 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.372915 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.375912 15493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:40.377956 15507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.378888 15493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:40.378988 15493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.379062 15493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.394420 15493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.394867 15493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.394999 15493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.395426 15493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.396170 15493 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.396214 15493 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.396258 15493 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.396286 15493 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.404814 15493 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:35077
I20240503 07:59:40.404851 15620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:35077 every 8 connection(s)
I20240503 07:59:40.405681 15493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 07:59:40.413223 15621 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.413417 15621 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.413859 15621 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.413975 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15493
I20240503 07:59:40.414068 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 07:59:40.414778 15170 ts_manager.cc:194] Registered new tserver with Master: 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:40.415359 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:33955
I20240503 07:59:40.415933 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:40.512537 15624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.512701 15624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.512741 15624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.515620 15624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.515676 15624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.515749 15624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 07:59:40.519114 15624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.519827 15624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.520821 15624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.528589 15632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.529271 15629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.529770 15630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.530444 15624 server_base.cc:1034] running on GCE node
I20240503 07:59:40.530764 15624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.531215 15624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.532351 15624 hybrid_clock.cc:648] HybridClock initialized: now 1714723180532330 us; error 35 us; skew 500 ppm
I20240503 07:59:40.533950 15624 webserver.cc:460] Webserver started at http://127.0.99.132:33799/ using document root <none> and password file <none>
I20240503 07:59:40.534426 15624 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.534480 15624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.534626 15624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.536042 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fed03e5364d24fb5b093f3a2007db5bf"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.536496 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fed03e5364d24fb5b093f3a2007db5bf"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.539395 15624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:40.541256 15638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.542104 15624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:40.542186 15624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fed03e5364d24fb5b093f3a2007db5bf"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.542248 15624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.554749 15624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.555163 15624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.555289 15624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.555708 15624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.556414 15624 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.556456 15624 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.556489 15624 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.556509 15624 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.564186 15624 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:41787
I20240503 07:59:40.564252 15751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:41787 every 8 connection(s)
I20240503 07:59:40.564953 15624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 07:59:40.570329 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15624
I20240503 07:59:40.570461 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 07:59:40.572362 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:40.572423 15752 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.572593 15752 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.573051 15752 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.573843 15170 ts_manager.cc:194] Registered new tserver with Master: fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:40.574343 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:36563
W20240503 07:59:40.670469 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.670641 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.670696 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.673573 15755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.673628 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.673704 15755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:40.677083 15755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.677780 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.678738 15755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.686434 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.687091 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.687788 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.688493 15755 server_base.cc:1034] running on GCE node
I20240503 07:59:40.688827 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.689283 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.690431 15755 hybrid_clock.cc:648] HybridClock initialized: now 1714723180690420 us; error 30 us; skew 500 ppm
I20240503 07:59:40.692046 15755 webserver.cc:460] Webserver started at http://127.0.99.133:41057/ using document root <none> and password file <none>
I20240503 07:59:40.692543 15755 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.692595 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.692739 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.694191 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "01ef5ba7dadb4fada6a034da2b2a4ed9"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.694608 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "01ef5ba7dadb4fada6a034da2b2a4ed9"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.697504 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 07:59:40.699416 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.700376 15755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:40.700475 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "01ef5ba7dadb4fada6a034da2b2a4ed9"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.700547 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.713287 15755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.713757 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.713883 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.714308 15755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.715047 15755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.715096 15755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.715142 15755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.715167 15755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.723155 15755 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34725
I20240503 07:59:40.723186 15882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34725 every 8 connection(s)
I20240503 07:59:40.723891 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:40.726897 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15755
I20240503 07:59:40.727011 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 07:59:40.729195 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:40.732143 15883 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.732362 15883 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.732967 15883 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.733937 15170 ts_manager.cc:194] Registered new tserver with Master: 01ef5ba7dadb4fada6a034da2b2a4ed9 (127.0.99.133:34725)
I20240503 07:59:40.734606 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:37245
W20240503 07:59:40.829085 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:40.829259 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:40.829303 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:40.832209 15886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:40.832267 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:40.832355 15886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:40.835772 15886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.15886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:40.836428 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:40.837419 15886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:40.845773 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.846206 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:40.847076 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:40.847620 15886 server_base.cc:1034] running on GCE node
I20240503 07:59:40.847913 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:40.848369 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:40.849505 15886 hybrid_clock.cc:648] HybridClock initialized: now 1714723180849490 us; error 36 us; skew 500 ppm
I20240503 07:59:40.851176 15886 webserver.cc:460] Webserver started at http://127.0.99.134:40051/ using document root <none> and password file <none>
I20240503 07:59:40.851636 15886 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:40.851691 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:40.851835 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:40.853303 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e30553332eab416999c058e898a96583"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.853711 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e30553332eab416999c058e898a96583"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.856552 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 07:59:40.858479 15900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.859326 15886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:40.859417 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e30553332eab416999c058e898a96583"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:40.859498 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:40.883605 15886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:40.884099 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:40.884227 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:40.884660 15886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:40.885468 15886 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:40.885511 15886 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.885556 15886 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:40.885583 15886 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:40.893178 15886 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42415
I20240503 07:59:40.893210 16013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42415 every 8 connection(s)
I20240503 07:59:40.893965 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:40.902247 16014 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:40.902413 16014 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:40.902819 16014 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:40.903667 15170 ts_manager.cc:194] Registered new tserver with Master: e30553332eab416999c058e898a96583 (127.0.99.134:42415)
I20240503 07:59:40.904192 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 15886
I20240503 07:59:40.904263 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:49401
I20240503 07:59:40.904299 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 07:59:40.906430 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:41.004392 16017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:41.004555 16017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:41.004595 16017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:41.007476 16017 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:41.007529 16017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:41.007591 16017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:41.010882 16017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.16017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:41.011521 16017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:41.012409 16017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:41.023277 16025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:41.023994 16022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:41.024538 16023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:41.025352 16017 server_base.cc:1034] running on GCE node
I20240503 07:59:41.025718 16017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:41.026217 16017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:41.027374 16017 hybrid_clock.cc:648] HybridClock initialized: now 1714723181027367 us; error 32 us; skew 500 ppm
I20240503 07:59:41.029114 16017 webserver.cc:460] Webserver started at http://127.0.99.135:45665/ using document root <none> and password file <none>
I20240503 07:59:41.029593 16017 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:41.029652 16017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:41.029839 16017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:41.031302 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b9d7e31e95e44354a782125c135f6489"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.031728 16017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b9d7e31e95e44354a782125c135f6489"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.034758 16017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 07:59:41.036653 16031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.037532 16017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:41.037633 16017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b9d7e31e95e44354a782125c135f6489"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.037710 16017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:41.048940 16017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:41.049391 16017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:41.049515 16017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:41.049928 16017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:41.050650 16017 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:41.050693 16017 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.050737 16017 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:41.050763 16017 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.058285 16017 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38963
I20240503 07:59:41.058313 16144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38963 every 8 connection(s)
I20240503 07:59:41.059065 16017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:41.060966 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 16017
I20240503 07:59:41.061069 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 07:59:41.063190 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:41.067445 16145 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:41.067626 16145 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:41.068068 16145 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:41.068982 15170 ts_manager.cc:194] Registered new tserver with Master: b9d7e31e95e44354a782125c135f6489 (127.0.99.135:38963)
I20240503 07:59:41.069656 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:59577
I20240503 07:59:41.093647 15359 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
W20240503 07:59:41.161079 16148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:41.161257 16148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:41.161298 16148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:41.164235 16148 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:41.164286 16148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:41.164347 16148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:41.167701 16148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.16148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:41.168323 16148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:41.169231 16148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:41.176846 16153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:41.177345 16156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:41.178090 16154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:41.178746 16148 server_base.cc:1034] running on GCE node
I20240503 07:59:41.179036 16148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:41.179497 16148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:41.180629 16148 hybrid_clock.cc:648] HybridClock initialized: now 1714723181180619 us; error 30 us; skew 500 ppm
I20240503 07:59:41.182250 16148 webserver.cc:460] Webserver started at http://127.0.99.136:33801/ using document root <none> and password file <none>
I20240503 07:59:41.182739 16148 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:41.182791 16148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:41.182938 16148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 07:59:41.184319 16148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1743e6a4667d49b49fb5a1e133103d77"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.184734 16148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1743e6a4667d49b49fb5a1e133103d77"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.187594 16148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:41.189455 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.190276 16148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:41.190366 16148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1743e6a4667d49b49fb5a1e133103d77"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:41.190445 16148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:41.204707 16148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:41.205199 16148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:41.205338 16148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:41.205762 16148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:41.206511 16148 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:41.206554 16148 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.206599 16148 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:41.206625 16148 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:41.213903 16148 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41731
I20240503 07:59:41.213927 16275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41731 every 8 connection(s)
I20240503 07:59:41.214650 16148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:41.218215 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 16148
I20240503 07:59:41.218336 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 07:59:41.222644 16276 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:41.222805 16276 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:41.223206 16276 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:41.223996 15170 ts_manager.cc:194] Registered new tserver with Master: 1743e6a4667d49b49fb5a1e133103d77 (127.0.99.136:41731)
I20240503 07:59:41.224488 15170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:36647
I20240503 07:59:41.232254 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:59:41.240622 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 07:59:41.240705 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 15755
I20240503 07:59:41.246275 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 15886
I20240503 07:59:41.250818 15490 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:41.251669 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 16017
I20240503 07:59:41.255831 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu with pid 16148
I20240503 07:59:41.417042 15621 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:41.575897 15752 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:45.011605 15170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:45.034183 15423 tablet_service.cc:1467] Processing CreateTablet for tablet e5b5e889c6b142f99e8facf2941e2983 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.034191 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 8a95f1e683a94566927691769f5fda97 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.035123 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a95f1e683a94566927691769f5fda97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.035120 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b5e889c6b142f99e8facf2941e2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.035755 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 06be4032feee4d269cc79a4969cf778f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.036001 15686 tablet_service.cc:1467] Processing CreateTablet for tablet e5b5e889c6b142f99e8facf2941e2983 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.036449 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06be4032feee4d269cc79a4969cf778f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.036729 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b5e889c6b142f99e8facf2941e2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.041019 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 8a95f1e683a94566927691769f5fda97 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.041031 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 8a95f1e683a94566927691769f5fda97 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.041257 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a95f1e683a94566927691769f5fda97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.041267 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a95f1e683a94566927691769f5fda97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.042250 15292 tablet_service.cc:1467] Processing CreateTablet for tablet e5b5e889c6b142f99e8facf2941e2983 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.042475 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b5e889c6b142f99e8facf2941e2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.034183 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 06be4032feee4d269cc79a4969cf778f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.042665 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06be4032feee4d269cc79a4969cf778f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.043527 16298 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:45.044332 15293 tablet_service.cc:1467] Processing CreateTablet for tablet 06be4032feee4d269cc79a4969cf778f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a9aeb39123e74a0dbb327cd3976ea7cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.044567 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06be4032feee4d269cc79a4969cf778f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.044553 16296 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:45.044875 16297 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:45.045683 16298 tablet_bootstrap.cc:654] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.046372 16296 tablet_bootstrap.cc:654] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.046442 16298 log.cc:824] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:45.046623 16297 tablet_bootstrap.cc:654] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.047075 16296 log.cc:824] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:45.047981 16297 log.cc:824] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:45.047988 16300 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:45.048687 16296 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:45.048866 16296 ts_tablet_manager.cc:1367] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 07:59:45.049547 16297 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:45.049600 16298 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:45.049682 16297 ts_tablet_manager.cc:1367] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 07:59:45.049718 16300 tablet_bootstrap.cc:654] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.049741 16298 ts_tablet_manager.cc:1367] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 07:59:45.050369 16300 log.cc:824] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Log is configured to *not* fsync() on all Append() calls
I20240503 07:59:45.051803 16300 tablet_bootstrap.cc:492] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:45.051947 16300 ts_tablet_manager.cc:1367] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 07:59:45.052184 16296 raft_consensus.cc:348] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.052361 16296 raft_consensus.cc:374] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.052413 16296 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.053009 16298 raft_consensus.cc:348] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.053056 16296 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.053045 16297 raft_consensus.cc:348] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.053202 16297 raft_consensus.cc:374] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.053253 16297 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.053360 16298 raft_consensus.cc:374] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.053458 16298 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.053820 16296 ts_tablet_manager.cc:1398] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 07:59:45.053908 16298 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.053648 16297 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.054791 16298 ts_tablet_manager.cc:1398] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240503 07:59:45.055065 16297 ts_tablet_manager.cc:1398] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:45.055186 16298 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:45.055385 16297 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:45.055132 16300 raft_consensus.cc:348] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.056062 16300 raft_consensus.cc:374] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.056114 16300 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.056176 16297 tablet_bootstrap.cc:654] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.056442 16298 tablet_bootstrap.cc:654] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.056448 16300 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.057201 16300 ts_tablet_manager.cc:1398] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 07:59:45.057504 16300 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:45.057543 16298 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:45.057601 16298 ts_tablet_manager.cc:1367] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:45.058053 16298 raft_consensus.cc:348] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.058135 16298 raft_consensus.cc:374] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.058161 16298 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.058476 16300 tablet_bootstrap.cc:654] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.058657 16296 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:45.058486 16298 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.059319 16298 ts_tablet_manager.cc:1398] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:45.059509 16296 tablet_bootstrap.cc:654] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.059849 16300 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:45.059931 16300 ts_tablet_manager.cc:1367] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:45.060408 16300 raft_consensus.cc:348] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.060504 16300 raft_consensus.cc:374] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.060544 16300 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.060670 16300 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.060946 16300 ts_tablet_manager.cc:1398] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.061128 16297 tablet_bootstrap.cc:492] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:45.061427 16297 ts_tablet_manager.cc:1367] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:45.061951 16297 raft_consensus.cc:348] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.062108 16297 raft_consensus.cc:374] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.062181 16297 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.062366 16297 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.062431 16296 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:45.062495 16296 ts_tablet_manager.cc:1367] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:45.062712 16297 ts_tablet_manager.cc:1398] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.062999 16296 raft_consensus.cc:348] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.063117 16296 raft_consensus.cc:374] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.063170 16296 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.063306 16296 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.063454 16297 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:45.063657 16296 ts_tablet_manager.cc:1398] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.064416 16297 tablet_bootstrap.cc:654] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.065714 16297 tablet_bootstrap.cc:492] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:45.065785 16297 ts_tablet_manager.cc:1367] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:45.066110 16297 raft_consensus.cc:348] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.066186 16297 raft_consensus.cc:374] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.066215 16297 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.066320 16297 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.066519 16297 ts_tablet_manager.cc:1398] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 07:59:45.068526 15753 tablet.cc:2369] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 07:59:45.084075 15360 tablet.cc:2369] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:45.107012 16304 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.107172 16304 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.108124 16304 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:45.112270 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:45.112365 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:45.112541 15706 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.112612 15313 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.112954 15378 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.113188 16304 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.113260 16304 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.113294 16304 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.114163 16304 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.114482 16304 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:45.114800 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:45.114867 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:45.114923 15706 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.114972 15313 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.115922 15706 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.116200 15378 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.116302 15313 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.116358 16304 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.116562 16304 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:45.116904 16304 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.119801 15169 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:45.159651 15622 tablet.cc:2369] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:45.200075 16306 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.200217 16306 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.201299 16306 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:45.205794 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:45.205929 15706 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:45.205922 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:45.206223 15575 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:45.206235 15247 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.206470 16306 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.206533 16306 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.206571 16306 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.207418 16306 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.207791 16306 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:45.208065 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:45.208130 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:45.208187 15575 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.208222 15706 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.209152 15575 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:45.209153 15706 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:45.209429 15247 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.209601 16306 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.209781 16306 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:45.210078 16306 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:45.212801 15169 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:45.242868 15491 tablet.cc:2369] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 07:59:45.253715 16311 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.253805 16311 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.254710 16311 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:45.258729 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:45.258850 15313 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:45.258863 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:45.259055 15444 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:45.259160 15509 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:45.259393 16311 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.259450 16311 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.259486 16311 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.260532 16311 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.260887 16311 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:45.261150 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:45.261166 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:45.261235 15444 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.261260 15313 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.261958 15313 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:45.262022 15444 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:45.262228 15511 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:45.262423 16311 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.262502 16311 raft_consensus.cc:686] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:45.262743 16311 consensus_queue.cc:237] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.265511 15169 catalog_manager.cc:5484] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:45.408758 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.408983 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.411885 16325 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 07:59:45.487665 15313 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.487690 15444 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.488240 16316 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.488373 15706 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.489161 16311 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.489872 16316 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.491393 15705 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.491777 16306 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.493978 15312 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.494908 16311 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.495159 15575 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:45.496068 16320 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:45.501613 16356 mvcc.cc:205] Tried to move back new op lower bound from 7023506167751335936 to 7023506166241431552. Current Snapshot: MvccSnapshot[applied={T|T < 7023506167751335936}]
I20240503 07:59:45.502097 16358 mvcc.cc:205] Tried to move back new op lower bound from 7023506167751335936 to 7023506166241431552. Current Snapshot: MvccSnapshot[applied={T|T < 7023506167751335936}]
I20240503 07:59:45.508119 16357 mvcc.cc:205] Tried to move back new op lower bound from 7023506167750848512 to 7023506166838161408. Current Snapshot: MvccSnapshot[applied={T|T < 7023506167750848512}]
I20240503 07:59:45.524489 16365 mvcc.cc:205] Tried to move back new op lower bound from 7023506167751335936 to 7023506166241431552. Current Snapshot: MvccSnapshot[applied={T|T < 7023506167751335936}]
INSERT report
rows total: 2000
time total: 95.2035 ms
time per row: 0.0476017 ms
I20240503 07:59:45.551194 15170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:45.555753 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 09079b0f21ed4bb0977c4d839e29d0dd (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.555795 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 09079b0f21ed4bb0977c4d839e29d0dd (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.556028 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09079b0f21ed4bb0977c4d839e29d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.556028 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09079b0f21ed4bb0977c4d839e29d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.556120 15423 tablet_service.cc:1467] Processing CreateTablet for tablet 964e76fe1aae447eb961f20965b1f754 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.556388 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964e76fe1aae447eb961f20965b1f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.556684 15293 tablet_service.cc:1467] Processing CreateTablet for tablet 468e137cf48f44488189628cd8aee5e0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.556955 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468e137cf48f44488189628cd8aee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.558327 16296 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:45.558307 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 09079b0f21ed4bb0977c4d839e29d0dd (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.558477 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 964e76fe1aae447eb961f20965b1f754 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.558537 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09079b0f21ed4bb0977c4d839e29d0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.558646 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964e76fe1aae447eb961f20965b1f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.558907 16298 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:45.559129 16296 tablet_bootstrap.cc:654] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.559664 16298 tablet_bootstrap.cc:654] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.559836 16297 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:45.560400 16300 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:45.560734 16297 tablet_bootstrap.cc:654] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.560978 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 468e137cf48f44488189628cd8aee5e0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.561187 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468e137cf48f44488189628cd8aee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.561221 16300 tablet_bootstrap.cc:654] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.561234 16298 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:45.561306 16298 ts_tablet_manager.cc:1367] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:45.561822 16298 raft_consensus.cc:348] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.561946 16298 raft_consensus.cc:374] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.561981 16298 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.562089 16298 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.562250 16297 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:45.562317 16297 ts_tablet_manager.cc:1367] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:45.562332 16298 ts_tablet_manager.cc:1398] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.562706 16297 raft_consensus.cc:348] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.562803 16297 raft_consensus.cc:374] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.562832 16297 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.562891 16298 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:45.562932 16297 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.563174 16297 ts_tablet_manager.cc:1398] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.563668 16298 tablet_bootstrap.cc:654] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.563889 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 964e76fe1aae447eb961f20965b1f754 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.563958 16300 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:45.564021 16300 ts_tablet_manager.cc:1367] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:45.564082 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964e76fe1aae447eb961f20965b1f754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.564438 16300 raft_consensus.cc:348] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.564544 16300 raft_consensus.cc:374] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.564572 16300 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.564667 16300 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.564908 16300 ts_tablet_manager.cc:1398] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.565042 16300 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:45.565085 15423 tablet_service.cc:1467] Processing CreateTablet for tablet 468e137cf48f44488189628cd8aee5e0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=63dbcc766316450f80704f00cc5d3868]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:45.565305 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468e137cf48f44488189628cd8aee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:45.565804 16300 tablet_bootstrap.cc:654] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.566233 16298 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:45.566299 16298 ts_tablet_manager.cc:1367] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:45.566679 16298 raft_consensus.cc:348] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.566764 16298 raft_consensus.cc:374] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.566793 16298 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.566905 16298 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.567152 16298 ts_tablet_manager.cc:1398] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.567948 16296 tablet_bootstrap.cc:492] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:45.568019 16296 ts_tablet_manager.cc:1367] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:45.568533 16296 raft_consensus.cc:348] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.568639 16296 raft_consensus.cc:374] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.568653 16300 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:45.568672 16296 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.568710 16300 ts_tablet_manager.cc:1367] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:45.568797 16296 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.569036 16296 ts_tablet_manager.cc:1398] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.569125 16300 raft_consensus.cc:348] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.569196 16296 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:45.569211 16300 raft_consensus.cc:374] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.569240 16300 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.569346 16300 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.569559 16300 ts_tablet_manager.cc:1398] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.569983 16296 tablet_bootstrap.cc:654] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.570854 16296 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:45.570912 16296 ts_tablet_manager.cc:1367] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:45.571159 16298 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:45.571265 16296 raft_consensus.cc:348] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.571346 16296 raft_consensus.cc:374] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.571372 16296 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.571470 16296 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.571923 16298 tablet_bootstrap.cc:654] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.573143 16298 tablet_bootstrap.cc:492] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:45.573215 16298 ts_tablet_manager.cc:1367] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:45.573540 16298 raft_consensus.cc:348] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.572117 16296 ts_tablet_manager.cc:1398] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.573621 16298 raft_consensus.cc:374] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.573652 16298 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.573729 16296 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:45.573745 16298 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.573971 16298 ts_tablet_manager.cc:1398] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.574739 16296 tablet_bootstrap.cc:654] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:45.576283 16296 tablet_bootstrap.cc:492] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:45.576364 16296 ts_tablet_manager.cc:1367] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:45.576696 16296 raft_consensus.cc:348] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.576790 16296 raft_consensus.cc:374] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:45.576834 16296 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:45.576941 16296 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.577172 16296 ts_tablet_manager.cc:1398] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:45.670876 16316 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.670989 16316 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.671391 16316 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:45.671845 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:45.672003 15705 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.672279 15378 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.672480 16316 raft_consensus.cc:2781] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.672555 16316 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.672605 16316 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.673478 16316 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.673786 16316 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:45.674119 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:45.674234 15705 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.675029 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:45.675149 15575 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.675153 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:45.675283 15574 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.675578 15705 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.675813 15378 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.675957 16316 raft_consensus.cc:2781] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.676018 16316 raft_consensus.cc:686] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:45.676108 15574 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.676131 16316 consensus_queue.cc:237] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:45.678047 15169 catalog_manager.cc:5484] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:45.717470 16316 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.717563 16316 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.717885 16316 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:45.718223 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:45.718207 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:45.718344 15312 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.718355 15574 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:45.718612 15377 leader_election.cc:302] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:45.718768 16316 raft_consensus.cc:2781] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.718827 16316 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.718852 16316 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.719544 16316 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.719828 16316 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:45.720118 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:45.720118 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:45.720202 15312 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.720235 15574 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.720917 15312 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.720988 15574 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:45.721164 15378 leader_election.cc:302] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:45.721343 16316 raft_consensus.cc:2781] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.721414 16316 raft_consensus.cc:686] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:45.721535 16316 consensus_queue.cc:237] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:45.723120 15169 catalog_manager.cc:5484] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:45.826718 16364 raft_consensus.cc:482] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:45.826850 16364 raft_consensus.cc:504] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.827896 16364 leader_election.cc:288] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:45.831091 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:45.831153 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:45.831230 15574 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:45.831283 15444 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:45.831526 15639 leader_election.cc:302] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.831727 16364 raft_consensus.cc:2781] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:45.831784 16364 raft_consensus.cc:482] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:45.831809 16364 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.832541 16364 raft_consensus.cc:504] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.832866 16364 leader_election.cc:288] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:45.833148 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:45.833236 15444 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.833978 15444 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:45.834175 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:45.834272 15574 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:45.834266 15642 leader_election.cc:302] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:45.834431 16364 raft_consensus.cc:2781] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:45.834518 16364 raft_consensus.cc:686] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:45.834769 16364 consensus_queue.cc:237] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:45.835036 15574 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:45.836863 15169 catalog_manager.cc:5484] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:45.948626 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:45.948832 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:45.951642 16401 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 07:59:46.022269 15574 raft_consensus.cc:1261] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.022668 16316 consensus_queue.cc:1035] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.024492 15575 raft_consensus.cc:1261] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.024863 16316 consensus_queue.cc:1035] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.027573 15312 raft_consensus.cc:1261] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.028151 16304 consensus_queue.cc:1035] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.029167 15705 raft_consensus.cc:1261] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.032976 15575 raft_consensus.cc:1261] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.033071 15444 raft_consensus.cc:1261] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.033382 16316 consensus_queue.cc:1035] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.033463 16307 consensus_queue.cc:1035] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.033797 16307 consensus_queue.cc:1035] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 94.4907 ms
time per row: 0.0472453 ms
I20240503 07:59:46.089800 15169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:46.094745 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd9a8cfe24949debc36c83257263ad1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.094789 15293 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd9a8cfe24949debc36c83257263ad1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.095031 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd9a8cfe24949debc36c83257263ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.095041 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd9a8cfe24949debc36c83257263ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.095832 15291 tablet_service.cc:1467] Processing CreateTablet for tablet f09926775f4140d7a2a8bdb26abfe54a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.096061 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09926775f4140d7a2a8bdb26abfe54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.097163 15292 tablet_service.cc:1467] Processing CreateTablet for tablet 74cc2c49c6b94f50841c7edbb0598c33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.097244 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 74cc2c49c6b94f50841c7edbb0598c33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.097352 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cc2c49c6b94f50841c7edbb0598c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.097478 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cc2c49c6b94f50841c7edbb0598c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.098309 15424 tablet_service.cc:1467] Processing CreateTablet for tablet f09926775f4140d7a2a8bdb26abfe54a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.098534 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09926775f4140d7a2a8bdb26abfe54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.099061 15423 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd9a8cfe24949debc36c83257263ad1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.099256 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd9a8cfe24949debc36c83257263ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.099742 15685 tablet_service.cc:1467] Processing CreateTablet for tablet f09926775f4140d7a2a8bdb26abfe54a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.099928 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09926775f4140d7a2a8bdb26abfe54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.100939 16449 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:46.101655 16449 tablet_bootstrap.cc:654] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.102018 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 74cc2c49c6b94f50841c7edbb0598c33 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a45b8dc5d0f54ccebbc36b671ec59e16]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.102181 16451 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:46.102216 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74cc2c49c6b94f50841c7edbb0598c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.102835 16451 tablet_bootstrap.cc:654] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.103549 16449 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:46.103655 16449 ts_tablet_manager.cc:1367] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:46.103657 16450 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:46.104022 16449 raft_consensus.cc:348] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.104107 16449 raft_consensus.cc:374] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.104135 16449 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.104250 16449 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.104475 16450 tablet_bootstrap.cc:654] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.104488 16449 ts_tablet_manager.cc:1398] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:46.104626 16449 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:46.105113 16448 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:46.105397 16449 tablet_bootstrap.cc:654] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.105895 16448 tablet_bootstrap.cc:654] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.106176 16451 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:46.106247 16451 ts_tablet_manager.cc:1367] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:46.106652 16451 raft_consensus.cc:348] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.106731 16450 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:46.106804 16450 ts_tablet_manager.cc:1367] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:46.106807 16451 raft_consensus.cc:374] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.106851 16451 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.106981 16451 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.107195 16450 raft_consensus.cc:348] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.107282 16451 ts_tablet_manager.cc:1398] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.107299 16450 raft_consensus.cc:374] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.107331 16450 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.107479 16450 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.107427 16451 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:46.107789 16450 ts_tablet_manager.cc:1398] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.108294 16451 tablet_bootstrap.cc:654] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.108354 16448 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:46.108487 16448 ts_tablet_manager.cc:1367] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:46.108690 16449 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:46.108920 16448 raft_consensus.cc:348] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.109403 16449 ts_tablet_manager.cc:1367] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:46.109652 16448 raft_consensus.cc:374] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.109748 16448 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.109933 16448 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.109978 16449 raft_consensus.cc:348] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.110131 16449 raft_consensus.cc:374] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.110250 16448 ts_tablet_manager.cc:1398] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:46.110209 16449 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.110569 16449 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.110731 16448 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:46.110824 16449 ts_tablet_manager.cc:1398] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.111001 16449 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:46.111378 16448 tablet_bootstrap.cc:654] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.111840 16449 tablet_bootstrap.cc:654] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.112789 16451 tablet_bootstrap.cc:492] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:46.112892 16451 ts_tablet_manager.cc:1367] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 07:59:46.113283 16451 raft_consensus.cc:348] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.113428 16451 raft_consensus.cc:374] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.113516 16451 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.113665 16451 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.113934 16451 ts_tablet_manager.cc:1398] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.114140 16448 tablet_bootstrap.cc:492] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:46.114218 16448 ts_tablet_manager.cc:1367] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:46.114533 16449 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:46.114614 16449 ts_tablet_manager.cc:1367] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:46.114650 16448 raft_consensus.cc:348] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.114740 16448 raft_consensus.cc:374] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.114804 16448 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.114943 16448 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.114989 16449 raft_consensus.cc:348] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.115103 16449 raft_consensus.cc:374] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.115155 16449 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.115288 16448 ts_tablet_manager.cc:1398] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.115295 16449 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.115598 16449 ts_tablet_manager.cc:1398] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.115446 16448 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:46.117002 16448 tablet_bootstrap.cc:654] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.117941 16448 tablet_bootstrap.cc:492] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:46.118021 16448 ts_tablet_manager.cc:1367] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:46.118330 16448 raft_consensus.cc:348] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.118407 16448 raft_consensus.cc:374] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.118433 16448 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.118546 16448 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.118738 16448 ts_tablet_manager.cc:1398] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:46.222568 16304 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.222679 16304 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.223043 16304 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.223433 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:46.223445 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:46.223562 15705 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:46.223577 15312 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:46.223853 15378 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.224025 16304 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.224089 16304 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.224114 16304 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.224928 16304 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.225219 16304 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.225548 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:46.225603 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:46.225649 15312 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.225728 15705 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.226423 15705 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:46.226436 15312 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:46.226660 15378 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.226794 16304 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.226873 16304 raft_consensus.cc:686] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:46.227015 16304 consensus_queue.cc:237] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.228404 15169 catalog_manager.cc:5484] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:46.282088 16364 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.282212 16364 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.282711 16364 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.283174 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:46.283311 15444 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:46.283604 15642 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:46.283775 16364 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.283828 16364 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.283882 16364 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.284677 16364 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.285029 16364 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.285363 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:46.285494 15444 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.286222 15444 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:46.286355 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:46.286463 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:46.286479 15312 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:46.286758 15642 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:46.286917 16364 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.287009 16364 raft_consensus.cc:686] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:46.287142 16364 consensus_queue.cc:237] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.288832 15169 catalog_manager.cc:5484] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:46.332253 16311 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.332388 16311 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.332902 16311 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.333269 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:46.333411 15313 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.333673 15509 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.333837 16311 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.333894 16311 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.333951 16311 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.334766 16311 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.335076 16311 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.335335 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:46.335425 15313 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.336359 15313 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:46.336611 15509 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.336658 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:46.336714 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:46.336792 16311 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.336812 15705 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.336910 16311 raft_consensus.cc:686] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:46.337040 16311 consensus_queue.cc:237] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:46.338457 15169 catalog_manager.cc:5484] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:46.489365 16467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:46.489571 16467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:46.492424 16467 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 07:59:46.563683 15313 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.564030 15706 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.565182 15706 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.565872 15444 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.566751 16465 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.567754 15313 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.567898 16307 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.568718 15313 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.568874 15312 raft_consensus.cc:1261] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.569021 16465 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.570925 16307 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.571204 15705 raft_consensus.cc:1261] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:46.571859 16444 consensus_queue.cc:1035] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:46.572146 16444 consensus_queue.cc:1035] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.013 ms
time per row: 0.0530063 ms
I20240503 07:59:46.635233 15171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:46.640139 15292 tablet_service.cc:1467] Processing CreateTablet for tablet c41a556e8bd54ad88a38ef933cf76443 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.640239 15555 tablet_service.cc:1467] Processing CreateTablet for tablet c41a556e8bd54ad88a38ef933cf76443 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.640429 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a556e8bd54ad88a38ef933cf76443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.640530 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a556e8bd54ad88a38ef933cf76443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.640705 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 9579e24381dd4a909624688351db8f03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.640969 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9579e24381dd4a909624688351db8f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.641530 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 00d9d6231fc54c07ae77084649980ad1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.641783 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d9d6231fc54c07ae77084649980ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.642742 16524 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:46.642742 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 9579e24381dd4a909624688351db8f03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.642971 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9579e24381dd4a909624688351db8f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.643445 16524 tablet_bootstrap.cc:654] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.643733 16526 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:46.644143 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 9579e24381dd4a909624688351db8f03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.644352 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9579e24381dd4a909624688351db8f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.644438 16526 tablet_bootstrap.cc:654] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.645119 15423 tablet_service.cc:1467] Processing CreateTablet for tablet c41a556e8bd54ad88a38ef933cf76443 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.645393 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41a556e8bd54ad88a38ef933cf76443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.645560 16526 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:46.645628 16526 ts_tablet_manager.cc:1367] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:46.645766 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 00d9d6231fc54c07ae77084649980ad1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.645977 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d9d6231fc54c07ae77084649980ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.646034 16526 raft_consensus.cc:348] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.646126 16526 raft_consensus.cc:374] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.646152 16526 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.646270 16526 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.646534 16526 ts_tablet_manager.cc:1398] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.646680 16526 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:46.647038 16525 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:46.647439 16526 tablet_bootstrap.cc:654] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.647815 16525 tablet_bootstrap.cc:654] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.645128 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 00d9d6231fc54c07ae77084649980ad1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b406acccfc2746c9bc5fe2395833c77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:46.648150 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d9d6231fc54c07ae77084649980ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:46.648391 16524 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:46.648461 16524 ts_tablet_manager.cc:1367] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 07:59:46.648874 16524 raft_consensus.cc:348] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.648964 16524 raft_consensus.cc:374] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.648995 16524 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.649112 16524 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.649369 16524 ts_tablet_manager.cc:1398] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.649487 16525 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:46.649561 16525 ts_tablet_manager.cc:1367] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:46.649858 16524 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:46.649915 16525 raft_consensus.cc:348] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.650002 16525 raft_consensus.cc:374] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.650030 16525 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.650152 16525 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.650406 16525 ts_tablet_manager.cc:1398] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.650563 16525 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:46.650650 16524 tablet_bootstrap.cc:654] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.650800 16526 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:46.650882 16526 ts_tablet_manager.cc:1367] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:46.651211 16529 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:46.651258 16526 raft_consensus.cc:348] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.651362 16526 raft_consensus.cc:374] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.651397 16525 tablet_bootstrap.cc:654] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.651398 16526 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.651504 16526 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.651738 16526 ts_tablet_manager.cc:1398] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.652134 16529 tablet_bootstrap.cc:654] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.653146 16525 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:46.653224 16525 ts_tablet_manager.cc:1367] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:46.653723 16525 raft_consensus.cc:348] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.653760 16529 tablet_bootstrap.cc:492] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:46.653801 16525 raft_consensus.cc:374] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.653829 16525 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.653838 16529 ts_tablet_manager.cc:1367] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:46.653916 16524 tablet_bootstrap.cc:492] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:46.653932 16525 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.654032 16524 ts_tablet_manager.cc:1367] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:46.654207 16525 ts_tablet_manager.cc:1398] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:46.654393 16525 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:46.654342 16529 raft_consensus.cc:348] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.654433 16529 raft_consensus.cc:374] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.654431 16524 raft_consensus.cc:348] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.654497 16529 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.654551 16524 raft_consensus.cc:374] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.654596 16524 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.654640 16529 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.654719 16524 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.654997 16529 ts_tablet_manager.cc:1398] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.655030 16524 ts_tablet_manager.cc:1398] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.655171 16529 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:46.655591 16525 tablet_bootstrap.cc:654] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.655989 16529 tablet_bootstrap.cc:654] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:46.656495 16525 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:46.656569 16525 ts_tablet_manager.cc:1367] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:46.656983 16525 raft_consensus.cc:348] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.657076 16525 raft_consensus.cc:374] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.657106 16525 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.657204 16525 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.657452 16525 ts_tablet_manager.cc:1398] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.657977 16529 tablet_bootstrap.cc:492] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:46.658064 16529 ts_tablet_manager.cc:1367] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:46.658403 16529 raft_consensus.cc:348] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.658501 16529 raft_consensus.cc:374] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:46.658533 16529 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:46.658644 16529 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.658864 16529 ts_tablet_manager.cc:1398] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:46.699757 16311 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.699853 16311 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.700371 16311 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.700723 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:46.700717 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:46.700889 15705 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.700889 15444 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.701134 15511 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:46.701290 16311 raft_consensus.cc:2781] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.701357 16311 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.701383 16311 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.702069 16311 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.702358 16311 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.702641 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:46.702735 15705 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.702754 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:46.702865 15444 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.703408 15705 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:46.703552 15444 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:46.703661 15509 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:46.703823 16311 raft_consensus.cc:2781] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.703909 16311 raft_consensus.cc:686] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:46.704051 16311 consensus_queue.cc:237] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.705447 15166 catalog_manager.cc:5484] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:46.737946 16311 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.738029 16311 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.738319 16311 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.738664 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:46.738664 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:46.738786 15312 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.738786 15444 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:46.739064 15511 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:46.739249 16311 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.739303 16311 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.739329 16311 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.739997 16311 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.740270 16311 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:46.740594 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:46.740594 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:46.740713 15312 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.740713 15444 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.741600 15312 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:46.741600 15444 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:46.741900 15509 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.742069 16311 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.742153 16311 raft_consensus.cc:686] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:46.742288 16311 consensus_queue.cc:237] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:46.743690 15166 catalog_manager.cc:5484] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:46.795676 16508 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:46.795783 16508 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.796247 16508 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.796525 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:46.796635 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:46.796669 15574 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:46.796758 15705 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:46.796972 15246 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.797116 16508 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:46.797173 16508 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:46.797215 16508 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.797930 16508 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.798231 16508 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:46.798537 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:46.798532 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:46.798624 15574 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.798632 15705 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:46.799257 15574 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:46.799329 15705 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:46.799501 15246 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:46.799695 16508 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:46.799767 16508 raft_consensus.cc:686] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:46.799888 16508 consensus_queue.cc:237] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:46.801261 15166 catalog_manager.cc:5484] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:46.934266 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:46.934619 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:46.937959 16540 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 07:59:47.030429 15444 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.030975 16311 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.033211 15443 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.033954 16311 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.036950 15705 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.038182 16311 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.039659 15312 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.040194 16311 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.060438 15574 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.060999 15706 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.061486 16508 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.061834 16508 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 140.717 ms
time per row: 0.0703585 ms
I20240503 07:59:47.117878 15171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:47.122953 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec0f68443746bdbb34cb202a128e62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.122946 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec0f68443746bdbb34cb202a128e62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.123235 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec0f68443746bdbb34cb202a128e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.123237 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec0f68443746bdbb34cb202a128e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.123790 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 83ec0f68443746bdbb34cb202a128e62 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.124025 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ec0f68443746bdbb34cb202a128e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.125034 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 263f44adeac1445bb1c38948ceb4b203 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.125270 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263f44adeac1445bb1c38948ceb4b203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.125412 16529 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:47.125968 16525 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:47.126184 16529 tablet_bootstrap.cc:654] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.126721 16525 tablet_bootstrap.cc:654] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.127355 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 96a7a7715c3944c0a973563648a4c04e (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.127559 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a7a7715c3944c0a973563648a4c04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.127707 16524 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:47.127774 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 96a7a7715c3944c0a973563648a4c04e (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.128011 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a7a7715c3944c0a973563648a4c04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.128517 16524 tablet_bootstrap.cc:654] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.129292 15292 tablet_service.cc:1467] Processing CreateTablet for tablet 263f44adeac1445bb1c38948ceb4b203 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.129500 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263f44adeac1445bb1c38948ceb4b203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.129782 16529 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:47.129849 16529 ts_tablet_manager.cc:1367] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:47.130093 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 263f44adeac1445bb1c38948ceb4b203 (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.130198 16529 raft_consensus.cc:348] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.130288 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263f44adeac1445bb1c38948ceb4b203. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.130286 16529 raft_consensus.cc:374] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.130316 16529 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.130420 16529 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.130674 16529 ts_tablet_manager.cc:1398] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.130813 16529 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:47.131556 16529 tablet_bootstrap.cc:654] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.131971 16526 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:47.132663 16526 tablet_bootstrap.cc:654] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.133587 16526 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:47.133673 16526 ts_tablet_manager.cc:1367] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:47.134056 16526 raft_consensus.cc:348] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.134145 16526 raft_consensus.cc:374] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.134172 16526 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.134281 16526 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.134523 16526 ts_tablet_manager.cc:1398] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.134550 16529 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:47.134608 16529 ts_tablet_manager.cc:1367] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:47.134665 16526 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:47.134889 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 96a7a7715c3944c0a973563648a4c04e (DEFAULT_TABLE table=rebalance_test_table_4 [id=49e7ca20b81e4ef78fc68702834a98f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.134963 16529 raft_consensus.cc:348] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.135051 16529 raft_consensus.cc:374] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.135079 16529 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.135143 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96a7a7715c3944c0a973563648a4c04e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.135180 16529 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.135411 16529 ts_tablet_manager.cc:1398] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.135432 16526 tablet_bootstrap.cc:654] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.136214 16525 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:47.136267 16525 ts_tablet_manager.cc:1367] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240503 07:59:47.136557 16525 raft_consensus.cc:348] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.136620 16525 raft_consensus.cc:374] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.136638 16525 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.136729 16525 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.136996 16525 ts_tablet_manager.cc:1398] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.137149 16525 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:47.137853 16525 tablet_bootstrap.cc:654] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.139029 16525 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:47.139101 16525 ts_tablet_manager.cc:1367] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.139432 16525 raft_consensus.cc:348] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.139496 16525 raft_consensus.cc:374] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.139514 16525 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.139602 16525 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.139824 16525 ts_tablet_manager.cc:1398] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.140547 16526 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:47.140619 16526 ts_tablet_manager.cc:1367] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:47.141053 16526 raft_consensus.cc:348] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.141144 16526 raft_consensus.cc:374] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.141171 16526 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.141280 16526 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.141502 16526 ts_tablet_manager.cc:1398] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.142367 16524 tablet_bootstrap.cc:492] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:47.142444 16524 ts_tablet_manager.cc:1367] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.001s
I20240503 07:59:47.142828 16524 raft_consensus.cc:348] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.142915 16524 raft_consensus.cc:374] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.142944 16524 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.143062 16524 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.143266 16524 ts_tablet_manager.cc:1398] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:47.143399 16524 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:47.144126 16524 tablet_bootstrap.cc:654] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.147507 16524 tablet_bootstrap.cc:492] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:47.147578 16524 ts_tablet_manager.cc:1367] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 07:59:47.147891 16524 raft_consensus.cc:348] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.147979 16524 raft_consensus.cc:374] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.148006 16524 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.148125 16524 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.148342 16524 ts_tablet_manager.cc:1398] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.148478 16524 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:47.149164 16524 tablet_bootstrap.cc:654] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.150074 16524 tablet_bootstrap.cc:492] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:47.150288 16524 ts_tablet_manager.cc:1367] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:47.150655 16524 raft_consensus.cc:348] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.150730 16524 raft_consensus.cc:374] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.150758 16524 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.150892 16524 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.151177 16524 ts_tablet_manager.cc:1398] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:47.185616 16320 raft_consensus.cc:482] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.185715 16320 raft_consensus.cc:504] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.186159 16320 leader_election.cc:288] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.186707 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:47.186853 15574 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:47.187126 15246 leader_election.cc:302] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:47.187341 16320 raft_consensus.cc:2781] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.187382 16320 raft_consensus.cc:482] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.187414 16320 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.188477 16320 raft_consensus.cc:504] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.188872 16320 leader_election.cc:288] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.189281 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:47.189388 15574 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.190172 15574 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:47.190239 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:47.190279 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:47.190366 15443 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:47.190444 15246 leader_election.cc:302] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:47.190657 16320 raft_consensus.cc:2781] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.190742 16320 raft_consensus.cc:686] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:47.190867 16320 consensus_queue.cc:237] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.192509 15166 catalog_manager.cc:5484] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:47.273593 16444 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.273684 16444 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.274036 16444 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:47.274343 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:47.274432 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:47.274474 15705 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:47.274569 15313 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:47.274746 15378 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.274890 16444 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.274948 16444 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.274973 16444 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.275708 16444 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.275980 16444 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:47.276275 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:47.276315 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:47.276371 15705 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.276401 15313 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.277115 15313 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:47.277145 15705 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:47.277364 15378 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.277506 16444 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.277640 16444 raft_consensus.cc:686] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:47.277762 16444 consensus_queue.cc:237] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:47.279203 15166 catalog_manager.cc:5484] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:47.488265 16364 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.488410 16364 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.488747 16364 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.489028 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:47.489068 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:47.489166 15574 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:47.489197 15313 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:47.489418 15639 leader_election.cc:302] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.489579 16364 raft_consensus.cc:2781] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.489637 16364 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.489665 16364 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.490414 16364 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.490703 16364 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.490978 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:47.490983 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:47.491083 15313 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.491083 15574 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.491914 15574 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:47.491914 15313 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:47.492180 15640 leader_election.cc:302] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.492338 16364 raft_consensus.cc:2781] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.492475 16364 raft_consensus.cc:686] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:47.492595 16364 consensus_queue.cc:237] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.494171 15166 catalog_manager.cc:5484] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:47.667306 16444 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.672237 16444 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 07:59:47.687793 16594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:47.687975 16594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:47.690898 16594 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:47.702610 15444 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.703650 16508 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.707420 16508 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 07:59:47.766224 15574 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.766682 16364 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:47.774804 15313 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:47.775187 16364 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 90.5807 ms
time per row: 0.0452903 ms
I20240503 07:59:47.818495 15168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:47.823158 15424 tablet_service.cc:1467] Processing CreateTablet for tablet b399cf884c564b999fa07885e7bd6cb7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.823292 15685 tablet_service.cc:1467] Processing CreateTablet for tablet b399cf884c564b999fa07885e7bd6cb7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.823446 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b399cf884c564b999fa07885e7bd6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.823542 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b399cf884c564b999fa07885e7bd6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.824376 15553 tablet_service.cc:1467] Processing CreateTablet for tablet 70d8ae824d0d4f0796c23ba15fed9507 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.824644 15553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d8ae824d0d4f0796c23ba15fed9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.825037 15555 tablet_service.cc:1467] Processing CreateTablet for tablet b399cf884c564b999fa07885e7bd6cb7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.825263 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b399cf884c564b999fa07885e7bd6cb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.825377 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 9434218808f44973b24e54e839a30e7c (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.825553 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434218808f44973b24e54e839a30e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.826115 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 70d8ae824d0d4f0796c23ba15fed9507 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.826350 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d8ae824d0d4f0796c23ba15fed9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.827446 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 9434218808f44973b24e54e839a30e7c (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.827495 16663 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:47.827610 16664 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:47.827659 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434218808f44973b24e54e839a30e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.828285 16663 tablet_bootstrap.cc:654] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.828390 16664 tablet_bootstrap.cc:654] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.829187 15423 tablet_service.cc:1467] Processing CreateTablet for tablet 70d8ae824d0d4f0796c23ba15fed9507 (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.829277 16663 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:47.829300 16664 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:47.829355 16664 ts_tablet_manager.cc:1367] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.829349 16663 ts_tablet_manager.cc:1367] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.829416 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70d8ae824d0d4f0796c23ba15fed9507. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.829711 16665 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:47.829738 16663 raft_consensus.cc:348] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.829738 16664 raft_consensus.cc:348] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.829828 16664 raft_consensus.cc:374] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.829828 16663 raft_consensus.cc:374] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.829856 16664 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.829856 16663 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.829965 16664 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.829974 16663 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.830209 16663 ts_tablet_manager.cc:1398] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.830206 16664 ts_tablet_manager.cc:1398] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.830346 16664 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:47.830471 16665 tablet_bootstrap.cc:654] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.831084 16664 tablet_bootstrap.cc:654] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.831436 16665 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:47.831514 16665 ts_tablet_manager.cc:1367] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.831918 16665 raft_consensus.cc:348] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.831998 16664 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:47.832012 16665 raft_consensus.cc:374] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.832033 16663 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:47.832052 16665 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.832065 16664 ts_tablet_manager.cc:1367] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.832154 16665 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.832401 16665 ts_tablet_manager.cc:1398] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.832428 16664 raft_consensus.cc:348] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.832520 16664 raft_consensus.cc:374] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.832549 16664 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.832644 16664 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.832810 16663 tablet_bootstrap.cc:654] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.832888 16664 ts_tablet_manager.cc:1398] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.833021 16664 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:47.833572 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 9434218808f44973b24e54e839a30e7c (DEFAULT_TABLE table=rebalance_test_table_5 [id=797291f92f614251a3d919d2d2801a47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:47.833796 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9434218808f44973b24e54e839a30e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:47.833791 16664 tablet_bootstrap.cc:654] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.834812 16663 tablet_bootstrap.cc:492] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:47.834898 16663 ts_tablet_manager.cc:1367] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 07:59:47.834975 16664 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:47.835048 16664 ts_tablet_manager.cc:1367] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:47.835275 16663 raft_consensus.cc:348] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.835366 16663 raft_consensus.cc:374] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.835394 16663 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.835438 16664 raft_consensus.cc:348] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.835528 16664 raft_consensus.cc:374] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.835512 16663 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.835557 16664 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.835659 16664 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.835752 16663 ts_tablet_manager.cc:1398] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:47.835887 16664 ts_tablet_manager.cc:1398] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.836184 16663 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:47.836735 16662 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:47.837358 16575 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.837432 16575 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.837637 16662 tablet_bootstrap.cc:654] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.837708 16663 tablet_bootstrap.cc:654] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.837741 16575 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:47.837951 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:47.838049 15444 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:47.838068 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:47.838297 15511 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:47.838452 16575 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.838496 16575 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.838531 16575 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
W20240503 07:59:47.838570 15509 leader_election.cc:341] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:47.839380 16575 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.839687 16663 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:47.839722 16575 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:47.839761 16663 ts_tablet_manager.cc:1367] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240503 07:59:47.839933 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:47.840045 15444 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.840387 16663 raft_consensus.cc:348] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.840487 16663 raft_consensus.cc:374] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.840534 16663 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.840639 16663 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.840727 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:47.840817 16662 tablet_bootstrap.cc:492] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:47.840900 16662 ts_tablet_manager.cc:1367] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:47.840637 15444 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:47.841079 16663 ts_tablet_manager.cc:1398] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.841342 16662 raft_consensus.cc:348] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.841435 16662 raft_consensus.cc:374] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.841470 16662 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
W20240503 07:59:47.841562 15509 leader_election.cc:341] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:47.841574 16662 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.841929 16662 ts_tablet_manager.cc:1398] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.841892 15511 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:47.842061 16575 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.842232 16575 raft_consensus.cc:686] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:47.842356 16575 consensus_queue.cc:237] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:47.843091 16662 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:47.843762 16662 tablet_bootstrap.cc:654] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:47.845331 16662 tablet_bootstrap.cc:492] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:47.845463 16662 ts_tablet_manager.cc:1367] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:47.845705 15170 catalog_manager.cc:5484] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:47.846072 16662 raft_consensus.cc:348] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.846210 16662 raft_consensus.cc:374] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:47.846284 16662 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:47.846434 16662 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.846755 16662 ts_tablet_manager.cc:1398] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:47.941519 16592 raft_consensus.cc:482] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.941640 16592 raft_consensus.cc:504] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.942031 16592 leader_election.cc:288] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.942380 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:47.942386 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:47.942507 15574 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:47.942520 15444 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:47.942775 15639 leader_election.cc:302] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.942931 16592 raft_consensus.cc:2781] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.942986 16592 raft_consensus.cc:482] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.943027 16592 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.943751 16592 raft_consensus.cc:504] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.944046 16592 leader_election.cc:288] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:47.944365 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:47.944335 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:47.944464 15444 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.944464 15574 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.945261 15444 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:47.945518 15642 leader_election.cc:302] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:47.945676 16592 raft_consensus.cc:2781] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.945758 16592 raft_consensus.cc:686] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:47.945904 16592 consensus_queue.cc:237] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:47.947470 15163 catalog_manager.cc:5484] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:47.961359 15574 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:47.989718 16590 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:47.989812 16590 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.990130 16590 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:47.990433 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:47.990422 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:47.990536 15313 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:47.990537 15574 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:47.990769 15377 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:47.990908 16590 raft_consensus.cc:2781] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:47.990971 16590 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:47.991004 16590 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.991742 16590 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.992054 16590 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:47.992283 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:47.992389 15574 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.992401 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:47.992516 15313 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:47.993069 15574 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:47.993259 15313 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:47.993302 15377 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:47.993451 16590 raft_consensus.cc:2781] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:47.993544 16590 raft_consensus.cc:686] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:47.993659 16590 consensus_queue.cc:237] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:47.995081 15163 catalog_manager.cc:5484] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:48.131747 16679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:48.131968 16679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:48.135010 16679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 07:59:48.179476 16364 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.179594 16364 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.180064 16364 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:48.180572 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:48.180740 15444 raft_consensus.cc:2379] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf in current term 1: Already voted for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in this term.
I20240503 07:59:48.180904 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:48.181318 15639 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:48.181538 16364 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.182528 16364 raft_consensus.cc:2726] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 07:59:48.208278 15706 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.208642 16674 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.210227 15444 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.210570 16674 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.212347 15313 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.212713 16444 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.213516 15574 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.214609 16590 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.222630 16712 mvcc.cc:205] Tried to move back new op lower bound from 7023506178897854464 to 7023506177403453440. Current Snapshot: MvccSnapshot[applied={T|T < 7023506178897854464}]
I20240503 07:59:48.225473 16596 mvcc.cc:205] Tried to move back new op lower bound from 7023506178897854464 to 7023506177403453440. Current Snapshot: MvccSnapshot[applied={T|T < 7023506178897854464}]
I20240503 07:59:48.234253 15574 raft_consensus.cc:1261] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.234627 16364 consensus_queue.cc:1035] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.247942 15444 raft_consensus.cc:1261] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.248273 16364 consensus_queue.cc:1035] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.326 ms
time per row: 0.0501631 ms
I20240503 07:59:48.287856 15171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:48.292699 15291 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e885ee89c4ac8836c944c8065af0b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.292845 15424 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e885ee89c4ac8836c944c8065af0b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.293097 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e885ee89c4ac8836c944c8065af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.293100 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e885ee89c4ac8836c944c8065af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.293614 15685 tablet_service.cc:1467] Processing CreateTablet for tablet ec6e885ee89c4ac8836c944c8065af0b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.293687 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 78f510497d7442a9adff8c55290a9a9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.293866 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6e885ee89c4ac8836c944c8065af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.294478 15292 tablet_service.cc:1467] Processing CreateTablet for tablet 78f510497d7442a9adff8c55290a9a9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.294708 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f510497d7442a9adff8c55290a9a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.295106 16663 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:48.295135 15293 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9f5f863324415ad1198a0c85ec9cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.295356 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9f5f863324415ad1198a0c85ec9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.295536 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9f5f863324415ad1198a0c85ec9cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.295728 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9f5f863324415ad1198a0c85ec9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.295900 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f510497d7442a9adff8c55290a9a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.296253 16663 tablet_bootstrap.cc:654] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.296720 16662 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:48.297389 16662 tablet_bootstrap.cc:654] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.298024 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 48f9f5f863324415ad1198a0c85ec9cc (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.298058 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 78f510497d7442a9adff8c55290a9a9b (DEFAULT_TABLE table=rebalance_test_table_6 [id=83fae01f26c14eebbd9e05c0910cf818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.298228 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f9f5f863324415ad1198a0c85ec9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.298285 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f510497d7442a9adff8c55290a9a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.298768 16662 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:48.298856 16662 ts_tablet_manager.cc:1367] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:48.299232 16662 raft_consensus.cc:348] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.299324 16662 raft_consensus.cc:374] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.299358 16662 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.299481 16662 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.299721 16662 ts_tablet_manager.cc:1398] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.299899 16662 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:48.300091 16664 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:48.300331 16663 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:48.300405 16663 ts_tablet_manager.cc:1367] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 07:59:48.300853 16663 raft_consensus.cc:348] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.300920 16664 tablet_bootstrap.cc:654] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.300953 16663 raft_consensus.cc:374] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.300989 16663 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.301115 16663 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.301281 16662 tablet_bootstrap.cc:654] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.301405 16663 ts_tablet_manager.cc:1398] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.301556 16663 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:48.302567 16663 tablet_bootstrap.cc:654] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.302618 16662 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:48.302698 16662 ts_tablet_manager.cc:1367] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:48.303238 16664 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:48.303330 16664 ts_tablet_manager.cc:1367] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:48.303712 16664 raft_consensus.cc:348] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.303812 16664 raft_consensus.cc:374] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.303841 16664 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.303936 16664 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.303923 16662 raft_consensus.cc:348] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.304021 16662 raft_consensus.cc:374] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.304158 16662 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.304196 16664 ts_tablet_manager.cc:1398] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.304301 16662 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.304589 16662 ts_tablet_manager.cc:1398] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.304761 16662 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:48.305699 16662 tablet_bootstrap.cc:654] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.306707 16662 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:48.306778 16662 ts_tablet_manager.cc:1367] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:48.307019 16663 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:48.307102 16663 ts_tablet_manager.cc:1367] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 07:59:48.307163 16662 raft_consensus.cc:348] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.307251 16662 raft_consensus.cc:374] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.307276 16662 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.307395 16662 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.307478 16663 raft_consensus.cc:348] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.307634 16662 ts_tablet_manager.cc:1398] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.307664 16663 raft_consensus.cc:374] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.307695 16663 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.307803 16663 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.308451 16663 ts_tablet_manager.cc:1398] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.322635 16665 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:48.323330 16665 tablet_bootstrap.cc:654] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.324256 16665 tablet_bootstrap.cc:492] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:48.324340 16665 ts_tablet_manager.cc:1367] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.324654 16665 raft_consensus.cc:348] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.324743 16665 raft_consensus.cc:374] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.324812 16665 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.324946 16665 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.325217 16665 ts_tablet_manager.cc:1398] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.325423 16665 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:48.326215 16665 tablet_bootstrap.cc:654] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.326664 16715 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.326774 16715 raft_consensus.cc:504] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.327188 16665 tablet_bootstrap.cc:492] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:48.327231 16715 leader_election.cc:288] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:48.327260 16665 ts_tablet_manager.cc:1367] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.327466 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:48.327580 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:48.327680 15444 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:48.327674 16665 raft_consensus.cc:348] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.327751 16665 raft_consensus.cc:374] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.327778 16665 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.327878 16665 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.327966 15642 leader_election.cc:302] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.328099 16715 raft_consensus.cc:2781] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 07:59:48.328047 15640 leader_election.cc:341] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c770157b083143399bcf339b4081a23f (127.0.99.129:36571): Illegal state: must be running to vote when last-logged opid is not known
I20240503 07:59:48.328138 16715 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:48.328137 16665 ts_tablet_manager.cc:1398] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.328172 16715 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.328454 16665 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:48.328986 16715 raft_consensus.cc:504] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.329274 16715 leader_election.cc:288] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:48.329470 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:48.329562 15313 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.329610 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:48.329703 15444 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.329826 16665 tablet_bootstrap.cc:654] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.330369 15313 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:48.330536 15444 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:48.330775 15640 leader_election.cc:302] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.330979 16715 raft_consensus.cc:2781] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:48.331053 16715 raft_consensus.cc:686] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:48.331197 16715 consensus_queue.cc:237] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.331379 16665 tablet_bootstrap.cc:492] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:48.331439 16665 ts_tablet_manager.cc:1367] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:48.331761 16665 raft_consensus.cc:348] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.331867 16665 raft_consensus.cc:374] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.331905 16665 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.332015 16665 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.332262 16665 ts_tablet_manager.cc:1398] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.332574 15162 catalog_manager.cc:5484] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:48.389122 16707 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.389230 16707 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.389546 16707 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:48.389911 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:48.389935 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:48.390028 15313 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:48.390038 15706 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:48.390300 15378 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:48.390460 16707 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:48.390517 16707 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:48.390554 16707 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.391304 16707 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.391665 16707 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:48.391908 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:48.392055 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:48.392010 15312 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.392153 15706 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.392829 15706 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:48.392850 15312 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:48.393077 15378 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.393231 16707 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:48.393334 16707 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:48.393504 16707 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:48.395146 15171 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:48.618649 16655 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.618763 16655 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.619143 16655 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:48.619474 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:48.619479 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:48.619601 15706 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:48.619625 15574 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:48.619881 15247 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.620048 16655 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:48.620113 16655 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:48.620146 16655 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.620970 16655 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.621271 16655 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:48.621563 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:48.621591 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:48.621675 15574 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.621685 15706 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.622534 15574 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:48.622534 15706 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:48.622790 15247 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.622949 16655 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:48.623092 16655 raft_consensus.cc:686] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:48.623215 16655 consensus_queue.cc:237] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:48.624858 15171 catalog_manager.cc:5484] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:48.759413 16729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:48.759601 16729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:48.762616 16729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 07:59:48.776109 16707 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.779632 16590 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240503 07:59:48.835419 15706 raft_consensus.cc:1261] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.837251 15574 raft_consensus.cc:1261] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.837888 16655 consensus_queue.cc:1035] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.838485 16655 consensus_queue.cc:1035] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.841138 15444 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.841509 16592 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.841799 15312 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:48.845369 16592 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:48.858146 16771 mvcc.cc:205] Tried to move back new op lower bound from 7023506181476356096 to 7023506180601749504. Current Snapshot: MvccSnapshot[applied={T|T < 7023506181461176320}]
INSERT report
rows total: 2000
time total: 99.6288 ms
time per row: 0.0498144 ms
I20240503 07:59:48.902052 15171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:48.907168 15292 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9f43270b7418d8c19a0cb5f4ec348 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.907343 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 110d0ab39d7c4f2aaae8967d49c90d09 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.907369 15554 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9f43270b7418d8c19a0cb5f4ec348 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.907455 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9f43270b7418d8c19a0cb5f4ec348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.907575 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110d0ab39d7c4f2aaae8967d49c90d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.907580 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9f43270b7418d8c19a0cb5f4ec348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.909067 15424 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9f43270b7418d8c19a0cb5f4ec348 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.909289 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9f43270b7418d8c19a0cb5f4ec348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.909603 16792 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:48.910123 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 420c787bccf84d02b8bf554193e72918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.910267 16791 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:48.910336 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420c787bccf84d02b8bf554193e72918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.910434 16792 tablet_bootstrap.cc:654] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.911024 16791 tablet_bootstrap.cc:654] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.911260 16794 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:48.911995 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 420c787bccf84d02b8bf554193e72918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.910123 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 110d0ab39d7c4f2aaae8967d49c90d09 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.912199 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420c787bccf84d02b8bf554193e72918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.912248 16794 tablet_bootstrap.cc:654] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.912254 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110d0ab39d7c4f2aaae8967d49c90d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.912235 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 110d0ab39d7c4f2aaae8967d49c90d09 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.912461 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110d0ab39d7c4f2aaae8967d49c90d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.913244 16794 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:48.913312 16794 ts_tablet_manager.cc:1367] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:48.913705 16794 raft_consensus.cc:348] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.913794 16794 raft_consensus.cc:374] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.913823 16794 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.913940 16794 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.914062 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 420c787bccf84d02b8bf554193e72918 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d45b17e93b24a449954eaf881a0d3e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:48.914188 16794 ts_tablet_manager.cc:1398] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.914258 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420c787bccf84d02b8bf554193e72918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:48.914335 16794 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:48.914494 16791 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:48.914563 16791 ts_tablet_manager.cc:1367] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:48.914906 16791 raft_consensus.cc:348] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.914983 16791 raft_consensus.cc:374] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.915004 16791 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.915064 16794 tablet_bootstrap.cc:654] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.915095 16791 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.915318 16791 ts_tablet_manager.cc:1398] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:48.915427 16791 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:48.916033 16794 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:48.916100 16791 tablet_bootstrap.cc:654] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.916101 16794 ts_tablet_manager.cc:1367] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.916488 16794 raft_consensus.cc:348] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.916570 16794 raft_consensus.cc:374] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.916608 16794 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.916704 16794 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.917014 16794 ts_tablet_manager.cc:1398] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.917668 16797 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:48.917711 16792 tablet_bootstrap.cc:492] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:48.917775 16792 ts_tablet_manager.cc:1367] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:48.917963 16791 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:48.918033 16791 ts_tablet_manager.cc:1367] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 07:59:48.918152 16792 raft_consensus.cc:348] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.918273 16792 raft_consensus.cc:374] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.918309 16792 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.918454 16797 tablet_bootstrap.cc:654] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.918421 16792 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:48.918424 16791 raft_consensus.cc:348] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.918510 16791 raft_consensus.cc:374] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.918543 16791 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.918669 16792 ts_tablet_manager.cc:1398] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.918668 16791 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.918833 16792 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:48.919294 16791 ts_tablet_manager.cc:1398] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.919597 16792 tablet_bootstrap.cc:654] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.920490 16797 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:48.920572 16797 ts_tablet_manager.cc:1367] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 07:59:48.920962 16792 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:48.921041 16792 ts_tablet_manager.cc:1367] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.921190 16797 raft_consensus.cc:348] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.921284 16797 raft_consensus.cc:374] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.921315 16797 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.921435 16797 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.921475 16792 raft_consensus.cc:348] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.921558 16792 raft_consensus.cc:374] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.921586 16792 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.921685 16797 ts_tablet_manager.cc:1398] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.921722 16792 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.921833 16797 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:48.922029 16792 ts_tablet_manager.cc:1398] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.922165 16792 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:48.922628 16797 tablet_bootstrap.cc:654] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.922889 16792 tablet_bootstrap.cc:654] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:48.923748 16792 tablet_bootstrap.cc:492] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:48.923760 16797 tablet_bootstrap.cc:492] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:48.923823 16792 ts_tablet_manager.cc:1367] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.923823 16797 ts_tablet_manager.cc:1367] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.924193 16797 raft_consensus.cc:348] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.924193 16792 raft_consensus.cc:348] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.924284 16792 raft_consensus.cc:374] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.924319 16792 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.924407 16797 raft_consensus.cc:374] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:48.924479 16797 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:48.924481 16792 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.924741 16797 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.925125 16797 ts_tablet_manager.cc:1398] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:48.925454 16792 ts_tablet_manager.cc:1398] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:48.976181 16592 raft_consensus.cc:482] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:48.976295 16592 raft_consensus.cc:504] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.976716 16592 leader_election.cc:288] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:48.977036 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:48.977061 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:48.977161 15574 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:48.977180 15444 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:48.977452 15642 leader_election.cc:302] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.977602 16592 raft_consensus.cc:2781] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:48.977661 16592 raft_consensus.cc:482] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:48.977695 16592 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.978408 16592 raft_consensus.cc:504] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.978683 16592 leader_election.cc:288] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:48.978971 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:48.978982 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:48.979090 15444 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.979090 15574 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:48.979900 15574 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:48.979918 15444 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:48.980214 15642 leader_election.cc:302] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:48.980402 16592 raft_consensus.cc:2781] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:48.980476 16592 raft_consensus.cc:686] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:48.980623 16592 consensus_queue.cc:237] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:48.982329 15171 catalog_manager.cc:5484] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:49.038971 16590 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.039076 16590 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.039395 16590 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.039785 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:49.039785 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:49.039923 15313 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.039923 15574 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.040192 15377 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:49.040338 16590 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.040412 16590 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.040439 16590 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.041239 16590 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.041524 16590 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.041833 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:49.041843 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:49.041941 15313 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.041962 15574 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.042624 15313 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.042737 15574 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.042879 15378 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:49.043042 16590 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.043146 16590 raft_consensus.cc:686] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:49.043283 16590 consensus_queue.cc:237] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.045195 15171 catalog_manager.cc:5484] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:49.080533 16592 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.080644 16592 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.081001 16592 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.081327 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:49.081480 15574 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:49.081511 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:49.081648 15313 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 0.
I20240503 07:59:49.081765 15639 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.081966 16592 raft_consensus.cc:2781] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.082018 16592 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.082049 16592 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.082937 16592 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.083225 16592 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.083537 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:49.083536 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:49.083626 15574 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.083662 15312 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.084276 15574 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:49.084512 15639 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.084558 15312 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 1.
I20240503 07:59:49.084650 16592 raft_consensus.cc:2781] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.084720 16592 raft_consensus.cc:686] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:49.084870 16592 consensus_queue.cc:237] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.086582 15171 catalog_manager.cc:5484] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 1 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:49.205662 16807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:49.205859 16807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:49.208866 16807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 07:59:49.281126 15312 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.281646 16590 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.281878 15574 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.285588 15444 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.286152 16590 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.287029 16592 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.287722 15313 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.292398 16715 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.292932 15575 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.294075 16780 mvcc.cc:205] Tried to move back new op lower bound from 7023506183287566336 to 7023506182322720768. Current Snapshot: MvccSnapshot[applied={T|T < 7023506183287566336}]
I20240503 07:59:49.297852 16592 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.301055 15575 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.304350 16715 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 101.147 ms
time per row: 0.0505736 ms
I20240503 07:59:49.347425 15162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:49.352108 15424 tablet_service.cc:1467] Processing CreateTablet for tablet d1de740a3d10414d9d9f78279eea9200 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.352108 15291 tablet_service.cc:1467] Processing CreateTablet for tablet d1de740a3d10414d9d9f78279eea9200 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.352377 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1de740a3d10414d9d9f78279eea9200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.352399 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1de740a3d10414d9d9f78279eea9200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.354395 16791 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:49.354389 16794 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:49.354902 15554 tablet_service.cc:1467] Processing CreateTablet for tablet d65f3357b1604254b1b15674013ee9d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.355127 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f3357b1604254b1b15674013ee9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.355172 16791 tablet_bootstrap.cc:654] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.355172 16794 tablet_bootstrap.cc:654] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.355949 15686 tablet_service.cc:1467] Processing CreateTablet for tablet d1de740a3d10414d9d9f78279eea9200 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.356201 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1de740a3d10414d9d9f78279eea9200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.355958 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3b87677104bd8b86e39f4bd81fdad (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.356345 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3b87677104bd8b86e39f4bd81fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.356364 15424 tablet_service.cc:1467] Processing CreateTablet for tablet d65f3357b1604254b1b15674013ee9d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.356576 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f3357b1604254b1b15674013ee9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.356752 16791 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:49.356854 16791 ts_tablet_manager.cc:1367] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.357134 15555 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3b87677104bd8b86e39f4bd81fdad (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.357240 16791 raft_consensus.cc:348] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.357344 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3b87677104bd8b86e39f4bd81fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.357347 16791 raft_consensus.cc:374] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.357383 16791 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.357477 16791 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.357717 16791 ts_tablet_manager.cc:1398] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.358327 16797 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:49.358898 16792 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:49.359050 16797 tablet_bootstrap.cc:654] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.359501 16792 tablet_bootstrap.cc:654] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.360652 16792 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:49.360716 16792 ts_tablet_manager.cc:1367] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.360735 15292 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3b87677104bd8b86e39f4bd81fdad (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.360985 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3b87677104bd8b86e39f4bd81fdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.361125 16792 raft_consensus.cc:348] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.361212 16792 raft_consensus.cc:374] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.361244 16792 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.361349 16792 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.361583 16792 ts_tablet_manager.cc:1398] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.361711 16792 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:49.361894 16797 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:49.361963 16797 ts_tablet_manager.cc:1367] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:49.362326 16792 tablet_bootstrap.cc:654] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.362339 16797 raft_consensus.cc:348] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.362427 16797 raft_consensus.cc:374] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.362454 16797 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.362560 16797 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.362798 16797 ts_tablet_manager.cc:1398] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.362892 15291 tablet_service.cc:1467] Processing CreateTablet for tablet d65f3357b1604254b1b15674013ee9d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=7bfd33c36697419f82eb7dd926df07ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.362962 16797 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:49.363085 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65f3357b1604254b1b15674013ee9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.363251 16791 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:49.363708 16797 tablet_bootstrap.cc:654] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.364027 16791 tablet_bootstrap.cc:654] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.365170 16791 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:49.365252 16791 ts_tablet_manager.cc:1367] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:49.364802 16792 tablet_bootstrap.cc:492] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:49.365401 16792 ts_tablet_manager.cc:1367] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 07:59:49.365679 16791 raft_consensus.cc:348] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.365763 16791 raft_consensus.cc:374] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.365746 16792 raft_consensus.cc:348] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.365845 16792 raft_consensus.cc:374] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.365878 16792 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.365798 16791 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.365975 16792 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.366015 16791 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.366278 16792 ts_tablet_manager.cc:1398] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.366472 16794 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:49.366545 16794 ts_tablet_manager.cc:1367] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240503 07:59:49.366801 16791 ts_tablet_manager.cc:1398] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 07:59:49.366941 16791 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:49.366978 16794 raft_consensus.cc:348] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.367055 16794 raft_consensus.cc:374] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.367084 16794 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.367183 16794 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.367424 16794 ts_tablet_manager.cc:1398] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.367575 16794 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:49.367753 16791 tablet_bootstrap.cc:654] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.368340 16794 tablet_bootstrap.cc:654] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.369616 16794 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:49.369690 16794 ts_tablet_manager.cc:1367] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.370082 16794 raft_consensus.cc:348] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.370167 16794 raft_consensus.cc:374] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.370195 16794 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.370302 16794 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.370590 16794 ts_tablet_manager.cc:1398] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.371038 16797 tablet_bootstrap.cc:492] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:49.371114 16797 ts_tablet_manager.cc:1367] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 07:59:49.371659 16797 raft_consensus.cc:348] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.371765 16797 raft_consensus.cc:374] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.371798 16797 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.371950 16797 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.372184 16797 ts_tablet_manager.cc:1398] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.372754 16791 tablet_bootstrap.cc:492] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:49.372865 16791 ts_tablet_manager.cc:1367] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 07:59:49.373262 16791 raft_consensus.cc:348] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.373368 16791 raft_consensus.cc:374] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.373407 16791 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.373517 16791 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.373750 16791 ts_tablet_manager.cc:1398] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.375813 16655 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.375905 16655 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.376221 16655 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.376485 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:49.376509 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:49.376564 15706 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:49.376600 15573 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:49.376827 15247 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.376976 16655 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.377030 16655 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.377058 16655 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.377628 16655 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.377908 16655 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.378163 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:49.378175 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:49.378259 15573 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.378265 15706 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.378852 15706 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:49.378971 15573 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:49.379096 15247 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.379395 16655 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.379482 16655 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:49.379597 16655 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.381018 15171 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:49.409534 16707 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.409642 16707 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.410013 16707 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:49.410357 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:49.410411 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:49.410517 15313 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.410530 15706 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.410841 15378 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:49.411098 16707 raft_consensus.cc:2781] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.411152 16707 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.411193 16707 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.411973 16707 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.412299 16707 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:49.412665 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:49.412667 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:49.412798 15706 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.412808 15313 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.413506 15313 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.413678 15706 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.413786 15378 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:49.413977 16707 raft_consensus.cc:2781] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.414062 16707 raft_consensus.cc:686] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:49.414182 16707 consensus_queue.cc:237] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:49.415742 15171 catalog_manager.cc:5484] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:49.597481 16707 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.597592 16707 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.597951 16707 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.598263 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f" is_pre_election: true
I20240503 07:59:49.598299 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:49.598415 15313 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.598428 15573 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.598676 15377 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:49.598830 16707 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.598889 16707 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.598915 16707 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.599642 16707 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.599941 16707 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.600235 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:49.600237 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:49.600332 15573 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.600332 15313 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.601104 15313 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.601119 15573 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.601341 15377 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:49.601480 16707 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.601568 16707 raft_consensus.cc:686] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:49.601699 16707 consensus_queue.cc:237] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:49.603245 15171 catalog_manager.cc:5484] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:49.758430 16855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:49.758621 16855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:49.761687 16855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 07:59:49.824136 16707 consensus_queue.cc:1035] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.829936 16707 consensus_queue.cc:1035] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.839859 15573 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.840298 15313 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.840759 16707 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.840898 16590 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.843516 16655 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:49.844226 15705 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:49.844794 16727 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.105 ms
time per row: 0.0555524 ms
I20240503 07:59:49.911144 15162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37678:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 07:59:49.916049 15686 tablet_service.cc:1467] Processing CreateTablet for tablet c7f31aec2eb34a41b3f725da97775274 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.916066 15424 tablet_service.cc:1467] Processing CreateTablet for tablet c7f31aec2eb34a41b3f725da97775274 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.916173 15555 tablet_service.cc:1467] Processing CreateTablet for tablet c7f31aec2eb34a41b3f725da97775274 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.916334 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f31aec2eb34a41b3f725da97775274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.916369 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f31aec2eb34a41b3f725da97775274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.916383 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f31aec2eb34a41b3f725da97775274. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.916864 15554 tablet_service.cc:1467] Processing CreateTablet for tablet 973e048f97aa4fd19f8b38ef68a9ec84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.917093 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973e048f97aa4fd19f8b38ef68a9ec84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.918514 16917 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:49.918576 16916 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:49.919373 16916 tablet_bootstrap.cc:654] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.919369 16917 tablet_bootstrap.cc:654] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.919414 15685 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.919662 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.920042 15424 tablet_service.cc:1467] Processing CreateTablet for tablet 973e048f97aa4fd19f8b38ef68a9ec84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.920198 16918 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:49.920240 15424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973e048f97aa4fd19f8b38ef68a9ec84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.921064 16918 tablet_bootstrap.cc:654] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.921056 15423 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.921238 15423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.921986 16918 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:49.922060 16918 ts_tablet_manager.cc:1367] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.922443 16918 raft_consensus.cc:348] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.922535 16918 raft_consensus.cc:374] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.922561 16918 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.922689 16918 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.923008 16918 ts_tablet_manager.cc:1398] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.923204 16918 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:49.923229 15291 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.923470 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83ee2fb1964aa4b7c662377ab4b0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.923741 16916 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:49.923807 16916 ts_tablet_manager.cc:1367] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 07:59:49.923988 16918 tablet_bootstrap.cc:654] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.924186 16916 raft_consensus.cc:348] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.924284 16916 raft_consensus.cc:374] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.924314 16916 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.924443 16916 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.924708 16916 ts_tablet_manager.cc:1398] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.924968 16916 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:49.925128 16918 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:49.925205 16918 ts_tablet_manager.cc:1367] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.925648 16918 raft_consensus.cc:348] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.925709 16923 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Bootstrap starting.
I20240503 07:59:49.925756 16918 raft_consensus.cc:374] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.925788 16918 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.925837 16916 tablet_bootstrap.cc:654] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.925916 16918 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.926226 16918 ts_tablet_manager.cc:1398] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.926779 16923 tablet_bootstrap.cc:654] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.927888 16923 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: No bootstrap required, opened a new log
I20240503 07:59:49.927975 16923 ts_tablet_manager.cc:1367] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:49.928200 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 973e048f97aa4fd19f8b38ef68a9ec84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=120cc6f7bb9c43bca39c66a1cb7e3007]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 07:59:49.928397 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973e048f97aa4fd19f8b38ef68a9ec84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 07:59:49.928472 16916 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:49.928414 16923 raft_consensus.cc:348] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.928527 16923 raft_consensus.cc:374] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.928558 16923 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.928561 16916 ts_tablet_manager.cc:1367] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 07:59:49.928961 16916 raft_consensus.cc:348] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.929049 16916 raft_consensus.cc:374] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.929075 16916 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.929162 16916 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.929411 16916 ts_tablet_manager.cc:1398] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.928671 16923 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.929572 16916 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064: Bootstrap starting.
I20240503 07:59:49.929705 16923 ts_tablet_manager.cc:1398] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:49.930317 16916 tablet_bootstrap.cc:654] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.930598 16918 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf: Bootstrap starting.
I20240503 07:59:49.931355 16918 tablet_bootstrap.cc:654] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.931766 16916 tablet_bootstrap.cc:492] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064: No bootstrap required, opened a new log
I20240503 07:59:49.931847 16916 ts_tablet_manager.cc:1367] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:49.932248 16916 raft_consensus.cc:348] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.932355 16916 raft_consensus.cc:374] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.932387 16916 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.932509 16916 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:49.932607 16917 tablet_bootstrap.cc:492] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:49.932683 16917 ts_tablet_manager.cc:1367] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240503 07:59:49.933147 16916 ts_tablet_manager.cc:1398] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 07:59:49.933131 16917 raft_consensus.cc:348] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.933233 16917 raft_consensus.cc:374] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.933276 16917 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.933385 16917 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.933609 16917 ts_tablet_manager.cc:1398] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 07:59:49.933759 16917 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275: Bootstrap starting.
I20240503 07:59:49.934159 16918 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf: No bootstrap required, opened a new log
I20240503 07:59:49.934244 16918 ts_tablet_manager.cc:1367] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 07:59:49.934618 16918 raft_consensus.cc:348] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.934701 16918 raft_consensus.cc:374] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.934734 16918 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.934790 16917 tablet_bootstrap.cc:654] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275: Neither blocks nor log segments found. Creating new log.
I20240503 07:59:49.934860 16918 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.935122 16918 ts_tablet_manager.cc:1398] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.936568 16917 tablet_bootstrap.cc:492] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275: No bootstrap required, opened a new log
I20240503 07:59:49.936650 16917 ts_tablet_manager.cc:1367] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 07:59:49.937435 16917 raft_consensus.cc:348] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.937530 16917 raft_consensus.cc:374] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 07:59:49.937558 16917 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Initialized, Role: FOLLOWER
I20240503 07:59:49.937810 16917 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.938030 16917 ts_tablet_manager.cc:1398] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 07:59:49.947784 16590 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:49.947876 16590 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.948210 16590 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.948488 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:49.948597 15706 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.948568 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" is_pre_election: true
I20240503 07:59:49.948715 15575 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 0.
I20240503 07:59:49.948854 15378 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.949008 16590 raft_consensus.cc:2781] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:49.949075 16590 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:49.949124 16590 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.949855 16590 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.950150 16590 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:49.950400 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:49.950402 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:49.950508 15706 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.950508 15575 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:49.951380 15575 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.951387 15706 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 1.
I20240503 07:59:49.951664 15378 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:49.951927 16590 raft_consensus.cc:2781] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:49.952055 16590 raft_consensus.cc:686] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:49.952219 16590 consensus_queue.cc:237] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:49.953662 15162 catalog_manager.cc:5484] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:50.126236 16839 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.126371 16839 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:50.126860 16839 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:50.127190 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:50.127352 15706 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:50.127322 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:50.127456 15444 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 0.
I20240503 07:59:50.127693 15509 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:50.127926 16839 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.127991 16839 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.128038 16839 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.129133 16839 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:50.129441 16839 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:50.129742 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:50.129729 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:50.129842 15444 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.129899 15706 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.130488 15444 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:50.130726 15511 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:50.130862 15706 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 1.
I20240503 07:59:50.130954 16839 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.131235 16839 raft_consensus.cc:686] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:50.131381 16839 consensus_queue.cc:237] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:50.133378 15162 catalog_manager.cc:5484] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 1 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:50.183677 16893 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 07:59:50.183785 16893 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:50.184185 16893 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:50.184489 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064" is_pre_election: true
I20240503 07:59:50.184615 15444 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:50.184579 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf" is_pre_election: true
I20240503 07:59:50.184725 15706 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 0.
I20240503 07:59:50.184929 15249 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:50.185184 16893 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 07:59:50.185243 16893 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 07:59:50.185284 16893 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.186034 16893 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:50.186349 16893 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:50.186633 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:50.186750 15444 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.187711 15444 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:50.187963 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:50.187991 15249 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:50.188086 15706 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 0 FOLLOWER]: Advancing to term 1
I20240503 07:59:50.188144 16893 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 07:59:50.188221 16893 raft_consensus.cc:686] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:50.188349 16893 consensus_queue.cc:237] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:50.189046 15706 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 1.
I20240503 07:59:50.190171 15162 catalog_manager.cc:5484] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 0 to 1, leader changed from <none> to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 1 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:50.320219 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.320418 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.323330 16933 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 07:59:50.423326 15706 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.423561 15442 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.423877 16930 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.424335 16930 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.434149 15706 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.434579 16893 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.438921 15442 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.439256 16893 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.440894 15705 raft_consensus.cc:1261] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.441329 16707 consensus_queue.cc:1035] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.441488 15575 raft_consensus.cc:1261] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 07:59:50.441834 16590 consensus_queue.cc:1035] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 07:59:50.448244 16961 mvcc.cc:205] Tried to move back new op lower bound from 7023506188015534080 to 7023506186045435904. Current Snapshot: MvccSnapshot[applied={T|T < 7023506188015534080}]
INSERT report
rows total: 2000
time total: 136.893 ms
time per row: 0.0684464 ms
I20240503 07:59:50.498188 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:34725
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:43357
--webserver_port=41057
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 07:59:50.598804 16991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.598986 16991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:50.599042 16991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.602089 16991 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:50.602152 16991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:50.602232 16991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 07:59:50.605683 16991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:34725
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=41057
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.16991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:50.606415 16991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:50.607383 16991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:50.615491 16999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:50.616155 16996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:50.616811 16991 server_base.cc:1034] running on GCE node
W20240503 07:59:50.617427 16997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:50.617789 16991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:50.618330 16991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:50.619490 16991 hybrid_clock.cc:648] HybridClock initialized: now 1714723190619473 us; error 33 us; skew 500 ppm
I20240503 07:59:50.626715 16991 webserver.cc:460] Webserver started at http://127.0.99.133:41057/ using document root <none> and password file <none>
I20240503 07:59:50.627231 16991 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:50.627300 16991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:50.630337 16991 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:50.631948 17005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.632867 16991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:50.632961 16991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "01ef5ba7dadb4fada6a034da2b2a4ed9"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:50.633333 16991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:50.656620 16991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:50.657135 16991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:50.657271 16991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:50.657730 16991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:50.658505 16991 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:50.658550 16991 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.658586 16991 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:50.658599 16991 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.666620 16991 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:34725
I20240503 07:59:50.666770 17118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:34725 every 8 connection(s)
I20240503 07:59:50.667523 16991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 07:59:50.672991 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 16991
I20240503 07:59:50.673341 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:42415
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:43357
--webserver_port=40051
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:50.675375 17119 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:50.675570 17119 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:50.676088 17119 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:50.677218 15169 ts_manager.cc:194] Re-registered known tserver with Master: 01ef5ba7dadb4fada6a034da2b2a4ed9 (127.0.99.133:34725)
I20240503 07:59:50.677794 15169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:48873
W20240503 07:59:50.777310 17122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.777487 17122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:50.777531 17122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.780570 17122 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:50.780627 17122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:50.780706 17122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 07:59:50.784384 17122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:42415
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=40051
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.17122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:50.785120 17122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:50.786084 17122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:50.793820 17127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:50.794425 17130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:50.795120 17128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:50.795843 17122 server_base.cc:1034] running on GCE node
I20240503 07:59:50.796154 17122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:50.796635 17122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:50.797777 17122 hybrid_clock.cc:648] HybridClock initialized: now 1714723190797767 us; error 39 us; skew 500 ppm
I20240503 07:59:50.799438 17122 webserver.cc:460] Webserver started at http://127.0.99.134:40051/ using document root <none> and password file <none>
I20240503 07:59:50.799916 17122 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:50.799978 17122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:50.803000 17122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:50.804618 17136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.805546 17122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:50.805634 17122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e30553332eab416999c058e898a96583"
format_stamp: "Formatted at 2024-05-03 07:59:40 on dist-test-slave-7tt0"
I20240503 07:59:50.805961 17122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:50.821218 17122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:50.821743 17122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:50.821898 17122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:50.822377 17122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:50.823109 17122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:50.823153 17122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.823191 17122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:50.823220 17122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.830668 17122 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:42415
I20240503 07:59:50.830718 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:42415 every 8 connection(s)
I20240503 07:59:50.831526 17122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 07:59:50.838618 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 17122
I20240503 07:59:50.839056 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:38963
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:43357
--webserver_port=45665
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:50.839648 17250 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:50.839849 17250 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:50.840378 17250 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:50.841291 15169 ts_manager.cc:194] Re-registered known tserver with Master: e30553332eab416999c058e898a96583 (127.0.99.134:42415)
I20240503 07:59:50.841955 15169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:33729
W20240503 07:59:50.942492 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:50.942662 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:50.942706 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:50.945709 17253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:50.945766 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:50.945837 17253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 07:59:50.949396 17253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:38963
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=45665
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.17253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:50.950114 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:50.951084 17253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:50.959146 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:50.959633 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:50.960559 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:50.961135 17253 server_base.cc:1034] running on GCE node
I20240503 07:59:50.961490 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:50.961983 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:50.963142 17253 hybrid_clock.cc:648] HybridClock initialized: now 1714723190963128 us; error 32 us; skew 500 ppm
I20240503 07:59:50.964809 17253 webserver.cc:460] Webserver started at http://127.0.99.135:45665/ using document root <none> and password file <none>
I20240503 07:59:50.965315 17253 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:50.965417 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:50.968458 17253 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 07:59:50.970227 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.971233 17253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 07:59:50.971324 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b9d7e31e95e44354a782125c135f6489"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:50.971729 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:50.991441 17253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:50.991875 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:50.991995 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:50.992401 17253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:50.993193 17253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:50.993238 17253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:50.993271 17253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:50.993291 17253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:51.001189 17253 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:38963
I20240503 07:59:51.001206 17380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:38963 every 8 connection(s)
I20240503 07:59:51.002169 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 07:59:51.004001 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 17253
I20240503 07:59:51.004410 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
/tmp/dist-test-taskfH3hjf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:41731
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:43357
--webserver_port=33801
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 07:59:51.011180 17381 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:51.011391 17381 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:51.011969 17381 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:51.013003 15169 ts_manager.cc:194] Re-registered known tserver with Master: b9d7e31e95e44354a782125c135f6489 (127.0.99.135:38963)
I20240503 07:59:51.013769 15169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:43909
W20240503 07:59:51.112471 17384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 07:59:51.112671 17384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 07:59:51.112721 17384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 07:59:51.115799 17384 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 07:59:51.115865 17384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 07:59:51.115957 17384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 07:59:51.119799 17384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:41731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=33801
--tserver_master_addrs=127.0.99.190:43357
--never_fsync=true
--heap_profile_path=/tmp/kudu.17384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 07:43:15 UTC on 1a15035a6830
build id 101
I20240503 07:59:51.120678 17384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 07:59:51.121779 17384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 07:59:51.130262 17390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:51.130790 17389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 07:59:51.131489 17392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 07:59:51.132095 17384 server_base.cc:1034] running on GCE node
I20240503 07:59:51.132421 17384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 07:59:51.132934 17384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 07:59:51.134079 17384 hybrid_clock.cc:648] HybridClock initialized: now 1714723191134068 us; error 24 us; skew 500 ppm
I20240503 07:59:51.136083 17384 webserver.cc:460] Webserver started at http://127.0.99.136:33801/ using document root <none> and password file <none>
I20240503 07:59:51.136548 17384 fs_manager.cc:359] Metadata directory not provided
I20240503 07:59:51.136605 17384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 07:59:51.139734 17384 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 07:59:51.141531 17398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:51.142427 17384 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 07:59:51.142513 17384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1743e6a4667d49b49fb5a1e133103d77"
format_stamp: "Formatted at 2024-05-03 07:59:41 on dist-test-slave-7tt0"
I20240503 07:59:51.142874 17384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 07:59:51.161623 17384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 07:59:51.162065 17384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 07:59:51.162186 17384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 07:59:51.162634 17384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 07:59:51.163388 17384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 07:59:51.163430 17384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:51.163465 17384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 07:59:51.163484 17384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 07:59:51.172184 17511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:41731 every 8 connection(s)
I20240503 07:59:51.172205 17384 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:41731
I20240503 07:59:51.173147 17384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfH3hjf/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714723112017232-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 07:59:51.179980 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfH3hjf/build/debug/bin/kudu as pid 17384
I20240503 07:59:51.180245 398 test_util.cc:268] Using random seed: 447925203
I20240503 07:59:51.181529 17512 heartbeater.cc:347] Connected to a master server at 127.0.99.190:43357
I20240503 07:59:51.181730 17512 heartbeater.cc:464] Registering TS with master...
I20240503 07:59:51.182305 17512 heartbeater.cc:511] Master 127.0.99.190:43357 requested a full tablet report, sending...
I20240503 07:59:51.183243 15169 ts_manager.cc:194] Re-registered known tserver with Master: 1743e6a4667d49b49fb5a1e133103d77 (127.0.99.136:41731)
I20240503 07:59:51.183950 15169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:35631
I20240503 07:59:51.191983 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 07:59:51.192193 398 test_util.cc:268] Using random seed: 447937150
I20240503 07:59:51.200323 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 07:59:51.200352 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.200529 398 test_util.cc:268] Using random seed: 447945487
I20240503 07:59:51.200532 15706 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.200603 15706 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.201589 15706 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.201946 15706 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.202364 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.202356 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.202472 15442 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.202473 15575 raft_consensus.cc:3032] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.202510 15575 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.202656 15575 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.202908 15575 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.203382 15442 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.203789 15642 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.203810 15575 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.203969 16884 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.204102 16884 raft_consensus.cc:686] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.204236 16884 consensus_queue.cc:237] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.204962 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.205080 15706 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.205137 15706 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.206239 15706 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.206810 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.206889 15575 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.207114 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.207135 15169 catalog_manager.cc:5484] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.207237 15312 raft_consensus.cc:3032] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.207273 15312 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.207433 15575 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.207417 15312 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.207556 15706 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.207657 15639 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.207648 15312 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.207998 16884 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.208065 16884 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.208166 16884 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.208878 15312 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.209415 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.209506 15706 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.209569 15706 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.210294 15706 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.210595 15706 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.210935 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.211022 15575 raft_consensus.cc:3032] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.211056 15575 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.211169 15575 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.211311 15575 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.211314 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.211414 15442 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.212072 15575 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.212163 15442 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.212530 15639 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.212695 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 07:59:51.212687 16884 raft_consensus.cc:2781] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.212755 16884 raft_consensus.cc:686] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.212836 398 test_util.cc:268] Using random seed: 447957795
I20240503 07:59:51.212893 16884 consensus_queue.cc:237] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.213462 15169 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.215292 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.215404 15706 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.215452 15706 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.216147 15706 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.216444 15706 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.216977 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.217084 15312 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.217417 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.217509 15706 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.217555 15706 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.217895 15312 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.218295 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.218436 15442 raft_consensus.cc:3032] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.218493 15442 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.218632 15442 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.218853 15442 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.218931 15169 catalog_manager.cc:5484] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.219659 15442 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.218218 15706 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.220865 15706 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.221107 15642 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.221271 17542 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.221366 17542 raft_consensus.cc:686] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.221477 17542 consensus_queue.cc:237] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.222295 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.222400 15575 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.223022 15575 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.223167 15169 catalog_manager.cc:5484] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.223309 15639 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.223455 17542 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.223520 17542 raft_consensus.cc:686] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.223623 17542 consensus_queue.cc:237] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.224426 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.224524 15313 raft_consensus.cc:3032] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.224575 15313 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.224695 15313 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.224876 15313 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.225718 15313 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.226202 15169 catalog_manager.cc:5484] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.227149 15706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.227265 15706 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.227330 15706 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.228092 15706 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.228449 15706 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.228955 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.229055 15442 raft_consensus.cc:3032] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.229096 15442 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.229241 15442 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.229410 15442 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.230170 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "420c787bccf84d02b8bf554193e72918"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.230211 15442 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.230273 15704 raft_consensus.cc:465] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.230525 15642 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.230671 16884 raft_consensus.cc:2781] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.230760 16884 raft_consensus.cc:686] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.230885 16884 consensus_queue.cc:237] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.231384 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.231487 15704 raft_consensus.cc:465] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.231565 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.231666 15312 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.232319 15169 catalog_manager.cc:5484] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.232409 15312 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.236120 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.236218 15704 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.236267 15704 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.237030 15704 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.237334 15704 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.237567 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.237658 15443 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.238462 15443 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.238781 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.238880 15573 raft_consensus.cc:3032] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.238925 15573 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.239061 15573 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.239212 15573 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.239472 15642 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.239630 16884 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.239701 16884 raft_consensus.cc:686] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.239820 16884 consensus_queue.cc:237] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.239995 15573 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.240278 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.240370 15704 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.240417 15704 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.242007 15169 catalog_manager.cc:5484] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.242070 15704 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.242364 15704 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.243289 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.243412 15312 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.243860 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.243953 15704 raft_consensus.cc:465] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.244231 15312 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.244665 15640 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.244807 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.244906 15443 raft_consensus.cc:3032] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.244951 15443 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.244979 17542 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.245110 17542 raft_consensus.cc:686] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.245114 15443 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.245266 15443 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.245239 17542 consensus_queue.cc:237] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.246006 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.246040 15443 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.246121 15704 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.246176 15704 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.246973 15704 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.247012 15169 catalog_manager.cc:5484] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.247344 15704 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.248389 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.248430 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a95f1e683a94566927691769f5fda97"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.248523 15704 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.248579 15704 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.248872 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.248961 15443 raft_consensus.cc:3032] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.248997 15443 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.249116 15443 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.249217 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 07:59:51.249244 15443 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.249369 398 test_util.cc:268] Using random seed: 447994327
I20240503 07:59:51.249348 15704 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.249693 15704 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.250027 15443 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.250274 15642 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:51.250279 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.250397 15312 raft_consensus.cc:3032] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.250443 15312 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.250475 16884 raft_consensus.cc:2781] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.250561 16884 raft_consensus.cc:686] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.250582 15312 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.250658 16884 consensus_queue.cc:237] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.250749 15312 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.251421 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.251524 15573 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.251572 15312 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.251951 15640 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.252197 17542 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.252277 17542 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.252364 15573 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.252410 17542 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.253048 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.253156 15704 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.253120 15169 catalog_manager.cc:5484] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.253218 15704 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.253948 15704 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.254520 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.254628 15573 raft_consensus.cc:3032] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.254678 15573 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.254814 15573 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.254925 15573 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.255678 15573 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.255999 15639 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.256170 16884 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.256099 15169 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.256232 16884 raft_consensus.cc:686] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.256336 16884 consensus_queue.cc:237] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.256589 15704 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.257122 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.257308 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.257417 15704 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.257464 15704 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.258194 15704 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.258793 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.258901 15575 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.258906 15169 catalog_manager.cc:5484] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.259265 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.259375 15442 raft_consensus.cc:3032] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.259421 15442 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.259550 15442 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.259689 15575 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.259711 15442 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.259985 15639 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.260147 17542 raft_consensus.cc:2781] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.260226 17542 raft_consensus.cc:686] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.260358 17542 consensus_queue.cc:237] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.260527 15442 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.261111 15704 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.261746 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9579e24381dd4a909624688351db8f03"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.261853 15704 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.261909 15704 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.262615 15704 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.262615 15169 catalog_manager.cc:5484] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.263156 15704 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.263140 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.263233 15575 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.263983 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.264025 15575 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.264150 15704 raft_consensus.cc:465] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.264719 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434218808f44973b24e54e839a30e7c"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.264912 15704 raft_consensus.cc:465] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.264966 15639 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.265179 16884 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.265264 16884 raft_consensus.cc:686] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.265292 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.265396 15313 raft_consensus.cc:3032] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.265434 15313 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.265409 16884 consensus_queue.cc:237] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.265549 15313 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.265687 15313 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.265726 15705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.265807 15705 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.265861 15705 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.266497 15313 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.266539 15705 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.266805 15705 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.267231 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.267315 15313 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.267318 15705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.267414 15705 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.267458 15705 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.268021 15313 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.268162 15705 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.268436 15705 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.268625 15640 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.268797 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.268833 17542 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.268903 17542 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.268883 15442 raft_consensus.cc:3032] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.268968 15442 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.269003 17542 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.269086 15442 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.269229 15442 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.269244 15169 catalog_manager.cc:5484] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.270125 15705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.270169 15442 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.270216 15705 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.270263 15705 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.270411 15642 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.270550 16884 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.270617 16884 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.270712 16884 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.270995 15705 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.271284 15705 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.271492 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.271580 15444 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.271570 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.271656 15313 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.272326 15444 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.272385 15313 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.272326 15169 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.272524 15169 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.272641 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.272738 15443 raft_consensus.cc:3032] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.272730 15642 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.272814 15443 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.272959 17542 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.272935 15443 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.273029 17542 raft_consensus.cc:686] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.273075 15443 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.273131 17542 consensus_queue.cc:237] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.273814 15443 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.274792 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.274902 15313 raft_consensus.cc:3032] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.274938 15313 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.275058 15313 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.275188 15313 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.275508 15169 catalog_manager.cc:5484] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 2 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.275915 15313 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 2.
I20240503 07:59:51.277043 15705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.277143 15705 raft_consensus.cc:465] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.277737 15705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:51.277828 15705 raft_consensus.cc:465] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Not starting forced leader election -- already a leader
W20240503 07:59:51.278551 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.280937 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.282590 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 07:59:51.282732 398 test_util.cc:268] Using random seed: 448027689
I20240503 07:59:51.285899 15575 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.286026 15443 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.286216 16884 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.286366 17542 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.297641 15573 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.297982 15313 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.298482 17542 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.301256 16884 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.303854 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.303978 15311 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.304030 15311 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.304913 15311 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.305253 15311 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.305927 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.306034 15574 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.306913 15574 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.307305 15246 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.307474 17618 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.308480 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.308586 15705 raft_consensus.cc:3032] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.308637 15705 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.308815 15705 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.309060 15705 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.309983 15705 raft_consensus.cc:2396] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.310964 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.311089 15313 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.311142 15313 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.311980 15313 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.312306 15313 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.313632 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.313738 15443 raft_consensus.cc:3032] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.313776 15443 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.313927 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.313910 15443 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.314014 15575 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.314062 15443 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.314793 15443 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.314793 15575 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.315529 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.315629 15312 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.315678 15312 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.315654 15246 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.315826 16893 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.316020 16893 raft_consensus.cc:686] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.316151 16893 consensus_queue.cc:237] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.316448 15312 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.316751 15312 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.318099 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.318202 15705 raft_consensus.cc:3032] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.318245 15705 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.318351 17618 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.318357 15705 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.318480 15705 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.318499 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.318565 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.318591 15444 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.318652 15311 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.318686 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_5 already exists
I20240503 07:59:51.318698 15311 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.318809 398 test_util.cc:268] Using random seed: 448063769
I20240503 07:59:51.319278 15705 raft_consensus.cc:2396] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.319443 15444 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.319445 15311 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.319686 17618 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.319774 15311 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.320294 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.320391 15573 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.320434 15249 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.321094 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.321187 15444 raft_consensus.cc:3032] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.321228 15444 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.321257 15573 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.321382 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.321353 15444 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.321465 15311 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.321517 15444 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.321527 15311 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.322275 15444 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.322275 15311 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.322578 15249 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.322609 15311 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.322866 16893 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.323020 16893 raft_consensus.cc:686] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.323146 16893 consensus_queue.cc:237] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.323712 16893 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.323793 16893 raft_consensus.cc:686] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.323925 16893 consensus_queue.cc:237] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.325137 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.325232 15312 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.325222 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.325279 15312 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.325191 15164 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.325304 15573 raft_consensus.cc:3032] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.325368 15573 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.325490 15573 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.325644 15573 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.325999 15312 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.326267 15312 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.326429 15573 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.326543 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.326639 15705 raft_consensus.cc:3032] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.326678 15705 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.326788 15705 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.326952 15705 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.327222 15246 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.327661 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.327764 15444 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.327775 15705 raft_consensus.cc:2396] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.327986 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.328075 15442 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.328614 15444 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
W20240503 07:59:51.329010 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.329485 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.329583 15312 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.329632 15312 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.329854 15442 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.330333 15312 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.330484 15249 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.330633 15312 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.330864 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.330950 15573 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.330952 16893 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.331037 16893 raft_consensus.cc:686] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.331171 16893 consensus_queue.cc:237] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.331193 15166 catalog_manager.cc:5484] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.331450 15166 catalog_manager.cc:5484] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.331669 15166 catalog_manager.cc:5484] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.331786 15573 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.331996 15246 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.331912 16893 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.332131 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.332247 15705 raft_consensus.cc:3032] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.332301 15705 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.332434 15705 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.332553 15705 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.332795 16893 raft_consensus.cc:686] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.333388 15705 raft_consensus.cc:2396] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.332931 16893 consensus_queue.cc:237] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.334437 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.334532 15312 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.334580 15312 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.335318 15312 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.335639 15312 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.335630 16893 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.335841 16893 raft_consensus.cc:686] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.335963 16893 consensus_queue.cc:237] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.336200 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.336297 15442 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.336383 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.336472 15312 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.336516 15312 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.337190 15442 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.336689 15166 catalog_manager.cc:5484] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.337255 15312 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.337905 15312 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.338277 15249 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.338917 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.338954 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.339012 15440 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.339071 15706 raft_consensus.cc:3032] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.339114 15706 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.339238 15706 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.339401 15706 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.339783 15440 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.340263 15706 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.340457 15249 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.340806 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.340896 15705 raft_consensus.cc:3032] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.340932 15705 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.341040 15705 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.341156 15705 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.341183 17618 raft_consensus.cc:2781] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.341255 17618 raft_consensus.cc:686] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.341372 17618 consensus_queue.cc:237] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.341929 15705 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.342396 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a95f1e683a94566927691769f5fda97"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.342494 15311 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.342546 15311 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.342747 17618 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.342811 17618 raft_consensus.cc:686] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.342932 17618 consensus_queue.cc:237] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.343341 15311 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.343426 15444 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.343654 15311 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.344310 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.344439 15574 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.345372 15574 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.345588 15572 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.345597 15312 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.345762 15573 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.346123 16884 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.346297 15246 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.346709 17637 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.347110 17569 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.347260 17637 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.347323 17637 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.347379 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.347476 15311 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.347434 17637 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.347550 15311 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.347944 15441 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.348125 17637 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.348469 15572 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.348791 17569 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.349848 15311 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.350193 15311 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.351157 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.351306 15311 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.351424 15311 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.352051 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.352191 15574 raft_consensus.cc:2379] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f in current term 2: Already voted for candidate fed03e5364d24fb5b093f3a2007db5bf in this term.
I20240503 07:59:51.352350 15311 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.353026 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.353163 15706 raft_consensus.cc:2379] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f in current term 2: Already voted for candidate fed03e5364d24fb5b093f3a2007db5bf in this term.
I20240503 07:59:51.353325 15311 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.353348 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.353444 15572 raft_consensus.cc:3032] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.353492 15572 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.353624 15572 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.353752 15572 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.354082 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.354233 15312 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.354352 15312 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.354616 15572 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.354846 16884 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.355268 15312 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.355787 15312 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.356088 15247 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c770157b083143399bcf339b4081a23f; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.356559 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.356667 15440 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.356608 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.356832 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.356927 15706 raft_consensus.cc:3032] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.356967 15706 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.356947 15311 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.357095 15706 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.357214 15706 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.357218 15311 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.357513 15440 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
W20240503 07:59:51.357650 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54926: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.358090 15706 raft_consensus.cc:2396] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.358255 15311 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.358408 15249 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.358834 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.358930 15705 raft_consensus.cc:3032] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.358983 15705 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.359098 15705 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.359120 16893 raft_consensus.cc:2726] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 07:59:51.359215 16893 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.359282 16893 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.359473 15311 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.359977 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.360090 15442 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.360320 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.360430 15440 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.360913 15442 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.359405 16893 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.361238 15249 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.361377 15440 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.361546 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.361625 15706 raft_consensus.cc:3032] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.361658 15706 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.361806 15705 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.361779 15706 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.361876 15249 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.361907 15706 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.362036 16893 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.361222 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.362241 15311 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.362354 15311 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.362655 15706 raft_consensus.cc:2396] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.362124 16893 raft_consensus.cc:686] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.362859 16893 consensus_queue.cc:237] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.363981 15705 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.363473 15311 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.364280 15166 catalog_manager.cc:5484] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.364549 15166 catalog_manager.cc:5484] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.364737 15166 catalog_manager.cc:5484] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.364989 15166 catalog_manager.cc:5484] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.365206 16893 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.365330 16893 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.365490 16893 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.365880 17565 mvcc.cc:205] Tried to move back new op lower bound from 7023506191755292672 to 7023506191319408640. Current Snapshot: MvccSnapshot[applied={T|T < 7023506191755292672 or (T in {7023506191755292672})}]
I20240503 07:59:51.366070 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.366171 15444 raft_consensus.cc:3032] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.366218 15444 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
W20240503 07:59:51.366361 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48458: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.366354 15444 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.366493 15311 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.366509 15444 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.367144 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.367257 15573 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.367172 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9579e24381dd4a909624688351db8f03"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.367316 15444 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.367437 15312 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.367542 15312 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.368098 15573 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.368366 15246 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.368425 15312 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.369168 17618 raft_consensus.cc:2781] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.369164 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.369243 17618 raft_consensus.cc:686] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.369264 15574 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.369359 17618 consensus_queue.cc:237] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.370098 15574 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.370415 15246 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.371655 17621 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.371737 17621 raft_consensus.cc:686] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.371881 17621 consensus_queue.cc:237] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.372089 15312 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.372362 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.372473 15705 raft_consensus.cc:3032] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.372521 15705 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.372658 15705 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.372790 15705 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.373618 15705 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.375687 15444 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.375916 15575 raft_consensus.cc:1226] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 2. Current term is 3. Ops: []
I20240503 07:59:51.376106 17618 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.376519 15705 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.377532 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.377624 15311 raft_consensus.cc:465] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 07:59:51.377871 17618 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.377954 17618 consensus_queue.cc:1225] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.378551 15705 pending_rounds.cc:76] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.378598 15705 pending_rounds.cc:97] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.378647 15705 raft_consensus.cc:2866] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.379326 15166 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.379581 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.379565 15166 catalog_manager.cc:5484] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.379676 15311 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.379722 15311 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.379760 15166 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.380026 15166 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:51.380147 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48472: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.380296 15166 catalog_manager.cc:5484] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.380499 15311 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.380561 15166 catalog_manager.cc:5484] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:51.381167 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55694: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.381314 15311 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.381467 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.381572 15575 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.381839 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.381935 15706 raft_consensus.cc:3032] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.381981 15706 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.382103 15706 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.382253 15706 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.382550 15575 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.382829 15246 leader_election.cc:302] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.383090 15706 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.384565 17618 raft_consensus.cc:2781] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.384632 17618 raft_consensus.cc:686] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.384735 17618 consensus_queue.cc:237] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.387979 15444 raft_consensus.cc:1261] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.388877 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.388972 15311 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.389021 15311 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.389660 15166 catalog_manager.cc:5484] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.389809 15311 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.390187 15311 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.395282 15575 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.395965 17554 mvcc.cc:205] Tried to move back new op lower bound from 7023506191755292672 to 7023506191319408640. Current Snapshot: MvccSnapshot[applied={T|T < 7023506191755292672}]
I20240503 07:59:51.396445 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.396529 15440 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.396550 15443 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.396590 17629 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.397117 17629 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.397532 15443 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 3.
I20240503 07:59:51.398722 15249 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.398988 16893 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.399058 16893 raft_consensus.cc:686] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.399176 16893 consensus_queue.cc:237] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.399214 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.399307 15706 raft_consensus.cc:3032] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.399355 15706 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.399470 15706 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.399608 15706 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.399902 16893 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.400481 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.400578 15706 raft_consensus.cc:2396] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.400578 15311 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.400625 15311 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.401530 15311 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.401962 15169 catalog_manager.cc:5484] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 3 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.402073 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.402180 15572 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.402325 15705 raft_consensus.cc:1261] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.403070 15572 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.403347 15246 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.403631 15311 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.405222 16893 raft_consensus.cc:2781] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.405303 16893 raft_consensus.cc:686] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.405426 16893 consensus_queue.cc:237] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.407481 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.407580 15705 raft_consensus.cc:3032] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.407627 15705 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.407742 15705 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.407881 15705 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.408550 15705 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.413640 15164 catalog_manager.cc:5484] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.414052 17621 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.414181 17621 consensus_queue.cc:1225] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.414750 15705 pending_rounds.cc:76] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.414800 15705 pending_rounds.cc:97] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.414834 15705 raft_consensus.cc:2866] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240503 07:59:51.417048 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48458: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:51.419751 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.427510 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_6 already exists
I20240503 07:59:51.427651 398 test_util.cc:268] Using random seed: 448172609
W20240503 07:59:51.427956 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.430330 15573 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 07:59:51.431479 17629 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:51.435770 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48472: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.436596 15705 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
W20240503 07:59:51.438338 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.439723 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48450: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.440424 16893 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.441293 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.441776 15311 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.441829 15311 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.442731 15311 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.443056 15311 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.443449 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
W20240503 07:59:51.443498 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55694: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.443554 15443 raft_consensus.cc:3032] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.443588 15443 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.443730 15443 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.443928 15443 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.443970 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.444070 15572 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.445058 15572 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.445438 15706 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.445991 15443 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.446290 15249 leader_election.cc:302] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.446465 17662 raft_consensus.cc:2781] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.446539 17662 raft_consensus.cc:686] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.446657 17662 consensus_queue.cc:237] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.447611 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_7 already exists
I20240503 07:59:51.447741 398 test_util.cc:268] Using random seed: 448192700
I20240503 07:59:51.449147 15574 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.449961 17662 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.450866 15169 catalog_manager.cc:5484] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.451846 17662 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.452680 15311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:51.452800 15311 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.452859 15311 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.453649 15311 raft_consensus.cc:504] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.454279 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.454608 15443 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.455410 15443 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.455629 15249 leader_election.cc:302] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.455907 17656 raft_consensus.cc:2781] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.455986 17656 raft_consensus.cc:686] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.456032 15440 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.456097 17656 consensus_queue.cc:237] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.456426 15440 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.456470 15572 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 07:59:51.456625 17637 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.456748 15573 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.456856 15311 leader_election.cc:288] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 2 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.456868 17637 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 07:59:51.457724 17629 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.457849 17618 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.458496 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.458598 15706 raft_consensus.cc:3032] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.458638 15706 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.458760 15706 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.458941 15706 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.459856 15706 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 2.
I20240503 07:59:51.460141 15703 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 07:59:51.460415 15702 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.461007 17662 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:51.461088 17629 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.461200 17629 consensus_queue.cc:1225] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.464849 15703 pending_rounds.cc:76] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.464898 15703 pending_rounds.cc:97] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.464931 15703 raft_consensus.cc:2866] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.469786 17557 mvcc.cc:205] Tried to move back new op lower bound from 7023506192197955584 to 7023506191864049664. Current Snapshot: MvccSnapshot[applied={T|T < 7023506192197955584 or (T in {7023506192197955584})}]
I20240503 07:59:51.478310 15163 catalog_manager.cc:5484] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 2 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:51.518919 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48492: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.527379 15705 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.527858 16893 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.528134 15442 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.532864 17662 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.559760 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_8 already exists
I20240503 07:59:51.559933 398 test_util.cc:268] Using random seed: 448304891
I20240503 07:59:51.577976 15444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.578117 15444 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.578181 15444 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.579006 15444 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.579288 15444 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.579599 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.579694 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.579731 15704 raft_consensus.cc:3032] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.579770 15704 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.579792 15575 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.579906 15704 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.580040 15704 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.580835 15575 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.580996 15704 raft_consensus.cc:2396] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.581602 15377 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.581840 16990 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.581930 16990 raft_consensus.cc:686] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.582055 16990 consensus_queue.cc:237] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.582954 15444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.583062 15444 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.583122 15444 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.583907 15444 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.584486 15444 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.584496 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.584599 15311 raft_consensus.cc:3032] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.584636 15311 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.584631 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.584733 15575 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.584751 15311 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.584904 15311 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.585371 15163 catalog_manager.cc:5484] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.585654 15575 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.585867 15311 raft_consensus.cc:2396] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.585954 15377 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.586311 17577 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.586407 17577 raft_consensus.cc:686] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.586525 17577 consensus_queue.cc:237] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.587651 15444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.587754 15444 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.587824 15444 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.588605 15444 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.588989 15163 catalog_manager.cc:5484] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.589239 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.589335 15311 raft_consensus.cc:3032] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.589377 15311 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.589488 15311 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.589618 15311 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.590448 15311 raft_consensus.cc:2396] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.590962 15444 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.591028 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.591121 15572 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.591779 15444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.591902 15444 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.591969 15444 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.592052 15572 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.592351 15377 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: c770157b083143399bcf339b4081a23f
I20240503 07:59:51.592520 16990 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.592624 16990 raft_consensus.cc:686] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.592744 16990 consensus_queue.cc:237] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.592859 15444 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.593403 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.593505 15311 raft_consensus.cc:3032] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.593554 15311 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.593686 15311 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.593833 15311 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.594561 15311 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.594849 15378 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.595009 16990 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.595093 16990 raft_consensus.cc:686] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.595213 16990 consensus_queue.cc:237] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.595894 15163 catalog_manager.cc:5484] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.596257 15444 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.597261 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.597378 15572 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.598135 15572 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.598670 15440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.598763 15440 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.598811 15440 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.599634 15440 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.599900 15440 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.600442 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.600528 15704 raft_consensus.cc:3032] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.600564 15704 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.600713 15704 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.600867 15704 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.601768 15704 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.600340 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.601954 15572 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.602062 15378 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.602216 17577 raft_consensus.cc:2781] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.602286 17577 raft_consensus.cc:686] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.602416 17577 consensus_queue.cc:237] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.602651 15572 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.604667 15163 catalog_manager.cc:5484] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.604972 15440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.605139 15440 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.605249 15440 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.605839 15273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.606571 15440 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.607338 15440 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.607221 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.607746 15704 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.608341 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.608441 15311 raft_consensus.cc:3032] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.608481 15311 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.608605 15311 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.608810 15311 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.609014 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.609630 15311 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
W20240503 07:59:51.609887 15273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.610672 15704 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.610965 15378 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.611122 16990 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.611193 16990 raft_consensus.cc:686] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.611308 16990 consensus_queue.cc:237] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.611498 15439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.611595 15439 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.611646 15439 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.612504 15439 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.612818 15439 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
W20240503 07:59:51.613272 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.613631 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.613714 15704 raft_consensus.cc:3032] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.613747 15704 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.613886 15704 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.614023 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.614126 15572 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.614174 15440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.614266 15440 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.614310 15440 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.615275 15163 catalog_manager.cc:5484] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.615514 15163 catalog_manager.cc:5484] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.614014 15704 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.617875 15704 raft_consensus.cc:2396] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.618204 15573 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240503 07:59:51.618273 15440 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.619203 16990 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:51.619937 15311 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240503 07:59:51.620417 16990 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240503 07:59:51.621343 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48450: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.623772 15440 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.624101 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.624190 15573 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.624738 15572 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.624890 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.624987 15310 raft_consensus.cc:3032] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.625030 15310 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.625032 15377 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.625082 15573 raft_consensus.cc:2396] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 07:59:51.625182 15310 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.625255 17577 raft_consensus.cc:2781] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.625335 15310 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.625334 17577 raft_consensus.cc:686] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.625443 17577 consensus_queue.cc:237] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.626310 15310 raft_consensus.cc:2396] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 7.
I20240503 07:59:51.626597 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.626704 15442 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.626749 15442 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.626789 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55694: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.627497 15442 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.627562 15378 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f
I20240503 07:59:51.627768 15442 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.628155 16590 raft_consensus.cc:2726] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 07:59:51.628648 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.628741 15442 raft_consensus.cc:482] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.628810 15442 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.629482 15442 raft_consensus.cc:504] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.629746 15442 leader_election.cc:288] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.630110 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.630213 15705 raft_consensus.cc:3032] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.630254 15705 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.630388 15705 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.630553 15705 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.631407 15705 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.631494 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.631583 15442 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.631633 15442 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.632184 15378 leader_election.cc:302] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.632346 17577 raft_consensus.cc:2781] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.632326 15442 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.632409 17577 raft_consensus.cc:686] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.632511 17577 consensus_queue.cc:237] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.632630 15442 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.633483 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.633582 15442 raft_consensus.cc:482] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.633634 15442 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.633821 15573 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.633854 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.633921 15574 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.634326 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.634335 15442 raft_consensus.cc:504] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.634351 16990 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.634416 15705 raft_consensus.cc:3032] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.634454 15705 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.634573 15705 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.634702 15705 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.634713 15574 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.635030 15442 leader_election.cc:288] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.635075 15573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.635171 15573 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.635552 15705 raft_consensus.cc:2396] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 5.
I20240503 07:59:51.635903 15573 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.636255 15377 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.636346 15312 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.636485 15377 leader_election.cc:302] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.636554 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.636636 15313 raft_consensus.cc:3032] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.636672 15313 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.636658 16990 raft_consensus.cc:2781] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.636734 16990 raft_consensus.cc:686] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.636822 15313 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.636869 16990 consensus_queue.cc:237] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.636979 15313 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.637333 16990 raft_consensus.cc:2781] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.637400 16990 raft_consensus.cc:686] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.637190 15163 catalog_manager.cc:5484] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.637492 16990 consensus_queue.cc:237] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.637827 15313 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.638465 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.638576 15311 raft_consensus.cc:3032] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.638625 15311 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.638756 15311 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.638895 15311 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.639768 15311 raft_consensus.cc:2396] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.640663 15575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.640759 15575 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.640688 15164 catalog_manager.cc:5484] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 1 to 2, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.640929 15164 catalog_manager.cc:5484] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.641135 15164 catalog_manager.cc:5484] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.641621 15575 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.641615 16590 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.641702 16590 consensus_queue.cc:1225] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Peer c770157b083143399bcf339b4081a23f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.642762 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.642860 15704 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.643447 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.643555 15442 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.643608 15442 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.643674 15704 raft_consensus.cc:2396] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.643985 15378 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064; no voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.644165 16990 raft_consensus.cc:2726] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:51.644335 15442 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.644387 15705 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240503 07:59:51.644644 15442 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.645215 16990 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:51.645386 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "420c787bccf84d02b8bf554193e72918"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.645480 15442 raft_consensus.cc:482] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.645532 15442 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.646351 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.646448 15705 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.647953 15311 pending_rounds.cc:76] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 3
I20240503 07:59:51.648005 15311 pending_rounds.cc:97] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.648061 15311 raft_consensus.cc:2866] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.648348 15705 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.648375 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.648476 15313 raft_consensus.cc:3032] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.648521 15313 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.648638 15313 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.648833 15313 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.648836 15575 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240503 07:59:51.649596 17577 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:51.649716 15313 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20240503 07:59:51.650115 15273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38988: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.650837 15378 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064; no voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.650965 17577 raft_consensus.cc:2726] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:51.655189 15442 raft_consensus.cc:504] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.656414 17581 mvcc.cc:205] Tried to move back new op lower bound from 7023506192961736704 to 7023506192804831232. Current Snapshot: MvccSnapshot[applied={T|T < 7023506192961736704}]
W20240503 07:59:51.657447 15273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.658092 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.658202 15704 raft_consensus.cc:3032] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.658252 15704 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.658396 15704 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.658572 15704 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.658751 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.658856 15571 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.659709 15573 raft_consensus.cc:1261] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.659767 15571 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.660164 17621 consensus_queue.cc:1035] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.660044 15377 leader_election.cc:302] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.660531 16590 raft_consensus.cc:2781] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.660600 16590 raft_consensus.cc:686] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.660707 16590 consensus_queue.cc:237] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
W20240503 07:59:51.660892 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48472: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.661645 15442 leader_election.cc:288] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.662621 15441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.662748 15441 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.662813 15441 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.663708 15441 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.664024 15441 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
W20240503 07:59:51.665002 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.665457 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.665551 15705 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.666415 15163 catalog_manager.cc:5484] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.668171 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.668292 15311 raft_consensus.cc:3032] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.668344 15311 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
W20240503 07:59:51.668493 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.668468 15311 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.668596 15311 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.669610 15311 raft_consensus.cc:2396] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20240503 07:59:51.669625 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55714: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.671413 15703 raft_consensus.cc:1261] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.671938 15704 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.673652 15570 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.674360 15705 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.674407 16893 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.675954 15378 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters: c770157b083143399bcf339b4081a23f
I20240503 07:59:51.676260 17577 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.676367 17577 raft_consensus.cc:686] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.676502 17577 consensus_queue.cc:237] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.677707 15440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.677827 15440 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.677887 15440 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.678781 15440 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.679383 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.679484 15705 raft_consensus.cc:3032] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.679523 15705 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.679646 15705 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.679795 15705 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.680016 17119 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:51.680683 15705 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.680959 15378 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.681133 16590 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.681200 16590 raft_consensus.cc:686] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.681310 16590 consensus_queue.cc:237] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.681704 15169 catalog_manager.cc:5484] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.682768 15440 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.683272 15569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.683379 15569 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.684170 15440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.684278 15440 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.684288 15569 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.684350 15440 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
W20240503 07:59:51.685225 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.685801 398 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_9 already exists
I20240503 07:59:51.685847 15440 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.686409 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.686512 15310 raft_consensus.cc:3032] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.686550 15310 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.686688 15310 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.686858 15310 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.687806 15310 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.689870 15169 catalog_manager.cc:5484] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.692217 15705 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.695101 15704 raft_consensus.cc:1261] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 07:59:51.694702 15575 raft_consensus.cc:1226] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 2. Current term is 3. Ops: []
W20240503 07:59:51.695557 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.696830 16590 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.698343 15310 raft_consensus.cc:1261] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 07:59:51.699157 15378 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.699322 16590 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.699395 16590 raft_consensus.cc:686] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.699503 16590 consensus_queue.cc:237] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.700146 15570 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 07:59:51.703439 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.703907 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38988: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.704502 17569 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.705338 15169 catalog_manager.cc:5484] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.706657 16893 consensus_queue.cc:1035] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.706753 16893 consensus_queue.cc:1225] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.707494 15311 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 07:59:51.707850 17542 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.707955 17662 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.712786 15570 raft_consensus.cc:1261] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 07:59:51.713218 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.713966 17577 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:51.716123 15570 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.717754 16590 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:51.720275 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.733008 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.736692 15310 raft_consensus.cc:1261] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 07:59:51.737206 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.737573 17706 consensus_queue.cc:1035] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:51.740237 15311 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240503 07:59:51.740525 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.741182 16590 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:51.743386 16590 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.743468 16590 consensus_queue.cc:1225] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Peer c770157b083143399bcf339b4081a23f log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.745365 15441 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240503 07:59:51.747794 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.749431 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.749982 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48488: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.750712 16893 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.753352 15705 pending_rounds.cc:76] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.753409 15705 pending_rounds.cc:97] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.753448 15705 raft_consensus.cc:2866] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240503 07:59:51.760066 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.760854 15310 pending_rounds.cc:76] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 3
I20240503 07:59:51.760905 15310 pending_rounds.cc:97] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.760936 15310 raft_consensus.cc:2866] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.763836 15703 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240503 07:59:51.763903 15273 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.766481 17653 mvcc.cc:205] Tried to move back new op lower bound from 7023506193264664576 to 7023506192765460480. Current Snapshot: MvccSnapshot[applied={T|T < 7023506193264664576 or (T in {7023506193264664576})}]
I20240503 07:59:51.767270 15703 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.767596 16590 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:51.768329 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.773149 15574 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 07:59:51.773479 17706 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:51.776286 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:51.777639 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.784273 17629 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.784385 17629 consensus_queue.cc:1225] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.785921 15705 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 07:59:51.785979 15705 pending_rounds.cc:76] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.786000 15705 pending_rounds.cc:97] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.786023 15705 raft_consensus.cc:2866] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240503 07:59:51.787846 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.792702 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.797555 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.799998 15705 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.800525 16990 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.800626 16990 consensus_queue.cc:1225] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Peer fed03e5364d24fb5b093f3a2007db5bf log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:51.802395 15575 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:51.804003 15704 pending_rounds.cc:76] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.804049 15704 pending_rounds.cc:97] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.804075 15704 raft_consensus.cc:2866] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.806437 15702 pending_rounds.cc:76] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Aborting all ops after (but not including) 3
I20240503 07:59:51.806480 15702 pending_rounds.cc:97] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 07:59:51.806505 15702 raft_consensus.cc:2866] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:51.806962 17584 mvcc.cc:205] Tried to move back new op lower bound from 7023506193384214528 to 7023506191972438016. Current Snapshot: MvccSnapshot[applied={T|T < 7023506193384214528}]
W20240503 07:59:51.809312 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:51.812835 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.815908 17706 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.816294 17629 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:51.823527 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.823681 15571 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.824750 15571 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
W20240503 07:59:51.826875 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.830338 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.838606 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39004: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.842015 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.843982 17250 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
W20240503 07:59:51.847465 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.849344 15571 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 07:59:51.850160 17662 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 07:59:51.851405 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.851639 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55694: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.856678 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:51.857676 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.818501 15440 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.866632 15441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.866750 15441 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.866807 15441 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.867154 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.867893 15441 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.868264 15441 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
W20240503 07:59:51.868633 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.869275 15441 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.869387 15441 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.869450 15441 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.870290 15441 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.870815 15443 raft_consensus.cc:1226] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 3. Current term is 4. Ops: []
I20240503 07:59:51.871027 15441 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.871412 17629 consensus_queue.cc:1046] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: INVALID_TERM, Last received: 3.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:51.871567 17618 raft_consensus.cc:3032] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.871626 17618 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.871755 17618 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.871910 17618 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.872855 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.872978 15704 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.873476 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.873879 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.873907 15704 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.874136 15378 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.874249 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.874289 17706 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.874378 17706 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.874349 15705 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.874536 17706 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.874536 15443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434218808f44973b24e54e839a30e7c"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.874691 15443 raft_consensus.cc:482] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.874938 15443 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.875409 15705 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.875746 15443 raft_consensus.cc:504] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.876063 15443 leader_election.cc:288] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.876924 15443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.877018 15443 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.877067 15443 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.877758 15443 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.878037 15443 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.879093 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.879184 15442 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.879231 15442 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.880052 15442 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.880311 15442 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:51.880569 15574 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240503 07:59:51.881148 15442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc"
dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
from {username='slave'} at 127.0.0.1:54948
I20240503 07:59:51.881247 15442 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.881301 15442 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.882002 15442 raft_consensus.cc:504] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.882043 15310 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240503 07:59:51.882277 15442 leader_election.cc:288] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.882611 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.882721 15310 raft_consensus.cc:3032] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.882772 15310 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
W20240503 07:59:51.882895 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.882927 15310 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.883124 15310 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.884073 15443 raft_consensus.cc:1226] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 2. Current term is 3. Ops: []
I20240503 07:59:51.884101 15310 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.884333 16590 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 07:59:51.884617 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.884677 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.884713 15702 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.884723 15378 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.884953 17577 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.885058 17577 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.885656 15702 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.885808 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.885946 15574 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.885879 17577 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.886044 15310 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 4.
I20240503 07:59:51.886135 15703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.886233 15703 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.886715 15171 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.886857 15574 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.887002 15703 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.887328 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:51.887429 15574 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
W20240503 07:59:51.888132 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.888697 15378 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.888888 15378 leader_election.cc:302] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.889015 17577 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.889078 17577 raft_consensus.cc:686] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.889178 17577 consensus_queue.cc:237] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.889607 17577 raft_consensus.cc:2781] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.889679 17577 raft_consensus.cc:686] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.889909 17577 consensus_queue.cc:237] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.889950 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.890048 15311 raft_consensus.cc:3032] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.890102 15311 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.890241 15311 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.7, Last appended by leader: 7, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.890362 15311 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:51.890401 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:51.890722 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.891276 15311 raft_consensus.cc:2396] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 4 because replica has last-logged OpId of term: 3 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20240503 07:59:51.891587 15443 raft_consensus.cc:1226] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 3. Current term is 4. Ops: [3.7-3.7]
I20240503 07:59:51.891826 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.891934 15311 raft_consensus.cc:3032] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.891984 15311 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
W20240503 07:59:51.892129 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.892113 15311 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.892243 15311 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.892899 17629 consensus_queue.cc:1046] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: INVALID_TERM, Last received: 2.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 07:59:51.892941 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48458: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.893097 17618 raft_consensus.cc:3032] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.893138 17618 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.893239 17618 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.893247 15311 raft_consensus.cc:2396] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.893360 17618 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.893597 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.893880 15703 raft_consensus.cc:1226] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 2. Current term is 3. Ops: [2.10-2.10]
I20240503 07:59:51.894785 15311 raft_consensus.cc:2396] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 07ae48a2fef84a10a186e6256e28c064 for term 3 because replica has last-logged OpId of term: 2 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11.
I20240503 07:59:51.894897 15574 raft_consensus.cc:2454] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 3.
I20240503 07:59:51.895692 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "07ae48a2fef84a10a186e6256e28c064" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.895788 15704 raft_consensus.cc:3032] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Stepping down as leader of term 1
I20240503 07:59:51.895774 15377 leader_election.cc:302] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.895833 15704 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.895969 15704 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.896114 15704 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 1 FOLLOWER]: Advancing to term 2
I20240503 07:59:51.896435 17577 raft_consensus.cc:2781] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 07:59:51.896571 17577 raft_consensus.cc:686] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.896692 17577 consensus_queue.cc:237] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.896935 15704 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07ae48a2fef84a10a186e6256e28c064 in term 2.
I20240503 07:59:51.884721 15705 raft_consensus.cc:1226] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 3. Current term is 4. Ops: [3.11-3.11]
I20240503 07:59:51.900354 15377 leader_election.cc:302] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.901029 17706 raft_consensus.cc:2781] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.901103 17706 raft_consensus.cc:686] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Leader. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
W20240503 07:59:51.901224 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.901212 17706 consensus_queue.cc:237] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.902614 15705 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20240503 07:59:51.903151 16590 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 07:59:51.904985 15163 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.905488 15163 catalog_manager.cc:5484] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 4 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.905730 15163 catalog_manager.cc:5484] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:51.905926 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.911460 17577 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 07:59:51.913671 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:51.914685 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.915135 15310 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20240503 07:59:51.915208 15163 catalog_manager.cc:5484] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 1 to 2, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 2 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.915462 15163 catalog_manager.cc:5484] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130). New cstate: current_term: 3 leader_uuid: "07ae48a2fef84a10a186e6256e28c064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.916644 17720 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:51.919965 17682 mvcc.cc:205] Tried to move back new op lower bound from 7023506193843965952 to 7023506193202507776. Current Snapshot: MvccSnapshot[applied={T|T < 7023506193843965952}]
W20240503 07:59:51.924427 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
W20240503 07:59:51.927654 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.936671 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.936853 15574 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.936923 15574 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.937927 15574 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.938721 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.938776 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.938840 15705 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.938892 15443 raft_consensus.cc:3032] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.938937 15443 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.939057 15443 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.939219 15443 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.939725 15705 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.939996 15509 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
W20240503 07:59:51.940029 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.940165 17561 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.940235 17561 raft_consensus.cc:686] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.940346 17561 consensus_queue.cc:237] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.940984 15443 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.941298 15574 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
W20240503 07:59:51.941478 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39016: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.941941 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.942052 15571 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.942111 15571 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.943269 15571 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.943614 15571 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.943933 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.944033 15310 raft_consensus.cc:3032] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.944072 15310 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.944200 15310 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.944584 15310 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.944628 15703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.944720 15703 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.944588 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "420c787bccf84d02b8bf554193e72918"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.945013 15574 raft_consensus.cc:482] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 07:59:51.945117 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.945186 15574 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.945559 15310 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.945614 15703 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.945837 15509 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.946130 17595 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.946209 17595 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.946329 17595 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.946415 15574 raft_consensus.cc:504] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.946756 15574 leader_election.cc:288] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.947285 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.947396 15443 raft_consensus.cc:3032] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.947446 15443 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.947582 15443 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.947722 15443 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.948024 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.948117 15702 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.948570 15443 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:51.948856 15511 leader_election.cc:302] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.948983 15702 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:51.949044 17561 raft_consensus.cc:2781] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.949131 17561 raft_consensus.cc:686] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.949249 17561 consensus_queue.cc:237] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.950182 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.950326 15574 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.950430 15574 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.951231 15574 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.952484 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.952549 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.952589 15311 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.952518 15574 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.952630 15705 raft_consensus.cc:3032] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.952662 15705 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:51.952818 15705 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.952970 15705 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.953420 15311 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:51.953649 15509 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:51.953708 15705 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:51.953801 17595 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.953889 17595 raft_consensus.cc:686] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.953998 17595 consensus_queue.cc:237] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.954038 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.954126 15570 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.954169 15570 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.955669 15570 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
W20240503 07:59:51.955765 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.956296 15570 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.956422 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.956521 15443 raft_consensus.cc:3032] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.956559 15443 raft_consensus.cc:729] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.956670 15443 consensus_queue.cc:260] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.956853 15443 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.956878 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.956964 15702 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.957664 15443 raft_consensus.cc:2396] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 5.
I20240503 07:59:51.957727 15702 raft_consensus.cc:2396] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 2 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 5.
I20240503 07:59:51.958362 15575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a95f1e683a94566927691769f5fda97"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.958436 15509 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:51.958459 15575 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.958510 15575 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.958616 17561 raft_consensus.cc:2726] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:51.959297 15575 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.959643 15575 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.960534 15572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.960633 15572 raft_consensus.cc:482] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.960678 15572 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.961217 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.961372 15311 raft_consensus.cc:3032] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.961427 15311 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.961459 15572 raft_consensus.cc:504] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.961551 15311 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.961773 15572 leader_election.cc:288] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.962605 15572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.962699 15572 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.962759 15572 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.962764 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.962877 15443 raft_consensus.cc:3032] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.962918 15443 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.963063 15443 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.963207 15443 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.963485 15572 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.963788 15572 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:51.963773 15703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.964006 15703 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.964521 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.964591 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.964690 15704 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.964870 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.964963 15705 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.965171 15703 raft_consensus.cc:2454] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
W20240503 07:59:51.965453 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54932: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.965610 15704 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:51.965734 15705 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.964715 15570 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 07:59:51.966208 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.966459 15443 raft_consensus.cc:2396] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.966478 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.966565 15310 raft_consensus.cc:3032] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:51.966609 15310 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:51.966087 15509 leader_election.cc:302] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.966674 15572 raft_consensus.cc:1226] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 2. Current term is 3. Ops: []
I20240503 07:59:51.966737 15310 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.14, Last appended by leader: 14, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.966828 15310 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:51.967033 15311 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.966753 15570 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.966898 15509 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.966874 17561 raft_consensus.cc:2781] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.968012 15509 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.968052 17561 raft_consensus.cc:686] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.968168 17561 consensus_queue.cc:237] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:51.968252 17595 raft_consensus.cc:2781] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:51.968421 17595 raft_consensus.cc:686] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.968549 17595 consensus_queue.cc:237] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:51.968717 17561 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.968803 17561 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.968910 17561 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.969568 15703 raft_consensus.cc:1226] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Rejecting Update request from peer c770157b083143399bcf339b4081a23f for earlier term 2. Current term is 3. Ops: [2.13-2.13]
I20240503 07:59:51.969547 15570 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.970094 15570 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.970130 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.956696 15163 catalog_manager.cc:5484] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.970224 15313 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.970479 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.970577 15443 raft_consensus.cc:3032] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.970623 15443 raft_consensus.cc:729] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.971063 15443 consensus_queue.cc:260] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.971123 15313 raft_consensus.cc:2396] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.971213 15443 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.971561 15311 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.972275 15443 raft_consensus.cc:2396] T 468e137cf48f44488189628cd8aee5e0 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:51.972551 15511 leader_election.cc:302] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f
I20240503 07:59:51.972630 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.972743 15570 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.972827 15570 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.972883 17728 raft_consensus.cc:2726] T 468e137cf48f44488189628cd8aee5e0 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:51.973532 15570 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.973860 15570 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.974152 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.974352 15311 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.974157 15310 raft_consensus.cc:2396] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 3 because replica has last-logged OpId of term: 2 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 13.
I20240503 07:59:51.976007 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.976107 15443 raft_consensus.cc:3032] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.976156 15443 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.976295 15311 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.976284 15443 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.976431 15443 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.976801 15509 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
W20240503 07:59:51.977020 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:51.977089 17729 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.977226 17729 raft_consensus.cc:686] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.977365 17729 consensus_queue.cc:237] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.978618 15443 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.979176 15163 catalog_manager.cc:5484] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.979415 15163 catalog_manager.cc:5484] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.979594 15163 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.979753 15163 catalog_manager.cc:5484] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.979926 15163 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.980101 15163 catalog_manager.cc:5484] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:51.980326 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.980427 15574 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.980484 15574 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.981866 15574 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.982434 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.982561 15443 raft_consensus.cc:3032] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.982614 15443 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.982736 15443 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.982880 15443 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.983867 15443 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.984112 15511 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:51.984272 15574 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.984683 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.984763 15311 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.985261 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.985400 15574 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.985482 15574 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.985596 15163 catalog_manager.cc:5484] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:51.986368 15574 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.986738 15574 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.987053 17730 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.987277 17730 raft_consensus.cc:686] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:51.987603 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.987699 15705 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.987515 17730 consensus_queue.cc:237] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.990235 15705 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.990358 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.990629 15571 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
W20240503 07:59:51.990795 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.991025 15571 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.991137 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.991412 15443 raft_consensus.cc:3032] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:51.991462 15443 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:51.991603 15509 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
W20240503 07:59:51.992055 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55710: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:51.991995 15443 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.992347 15443 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.993350 15443 raft_consensus.cc:2454] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
W20240503 07:59:51.993732 15272 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:51.994468 15311 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.994597 15313 raft_consensus.cc:1226] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 3. Current term is 4. Ops: []
I20240503 07:59:51.994830 15571 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:51.995175 15571 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:51.995807 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:51.995920 15570 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:51.995988 15570 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.996459 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:51.996563 15705 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.997212 15570 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:51.997391 15572 raft_consensus.cc:1226] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 3. Current term is 4. Ops: []
I20240503 07:59:51.998263 15705 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:51.998486 17720 consensus_queue.cc:1046] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: INVALID_TERM, Last received: 3.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 07:59:51.998716 15570 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:51.998888 15509 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:51.999035 17728 raft_consensus.cc:2781] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:51.999156 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:51.999250 15313 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:51.999285 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:51.999954 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.000037 15313 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.000056 15310 raft_consensus.cc:3032] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.000105 15310 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.000274 15509 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.000404 15443 raft_consensus.cc:3032] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.000427 17561 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.000485 17561 raft_consensus.cc:686] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.000494 15443 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.000594 17561 consensus_queue.cc:237] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.000627 15443 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.000739 15443 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.001101 17561 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.001163 17561 raft_consensus.cc:686] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.001253 17561 consensus_queue.cc:237] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.17, Last appended by leader: 17, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.001355 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.001449 15570 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.001500 15570 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.001664 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54926: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.001761 15443 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
W20240503 07:59:52.001849 16990 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20240503 07:59:52.002190 17652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54974: Illegal state: Replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } }
I20240503 07:59:52.002815 15570 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.003371 15570 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.003732 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.003825 15443 raft_consensus.cc:3032] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.003865 15443 raft_consensus.cc:729] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.004168 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.004397 15571 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.004451 15571 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.005836 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.006649 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.006670 15571 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.006752 15705 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.007229 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.007346 15442 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.007576 15705 raft_consensus.cc:2396] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:52.008137 15442 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.008395 15511 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275; no voters:
I20240503 07:59:52.008666 17729 raft_consensus.cc:2781] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.008844 17729 raft_consensus.cc:686] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.008983 17729 consensus_queue.cc:237] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
W20240503 07:59:52.009088 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.009585 15571 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.009908 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.010043 15313 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.010166 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9579e24381dd4a909624688351db8f03"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.010254 15571 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.010300 15571 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.010761 15443 consensus_queue.cc:260] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.010917 15443 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.011075 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Service unavailable: leader is not yet ready
I20240503 07:59:52.011090 15313 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.005791 15310 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.011740 15310 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.012389 15310 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
W20240503 07:59:52.013236 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.013389 17728 raft_consensus.cc:686] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.013548 17728 consensus_queue.cc:237] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.014098 15169 catalog_manager.cc:5484] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.015832 15443 raft_consensus.cc:2396] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:52.015969 17381 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:52.016115 15511 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275; no voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:52.017412 17730 raft_consensus.cc:2726] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240503 07:59:52.018446 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39004: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.018790 15169 catalog_manager.cc:5484] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.018983 15169 catalog_manager.cc:5484] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.019129 15169 catalog_manager.cc:5484] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.019251 15169 catalog_manager.cc:5484] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.021576 15443 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 07:59:52.021617 15313 raft_consensus.cc:1261] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 07:59:52.022046 17595 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 07:59:52.022557 17728 consensus_queue.cc:1035] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 07:59:52.026394 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.028445 15442 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20240503 07:59:52.028662 15311 raft_consensus.cc:1261] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 17. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20240503 07:59:52.028797 17561 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240503 07:59:52.029176 15443 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240503 07:59:52.030701 17561 consensus_queue.cc:1035] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20240503 07:59:52.031606 17561 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:52.033179 15313 raft_consensus.cc:1261] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240503 07:59:52.034125 15571 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.034453 15571 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.035389 17595 consensus_queue.cc:1035] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:52.035686 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.035775 15311 raft_consensus.cc:3032] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.035810 15311 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.035934 15311 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 3.22, Last appended by leader: 22, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.036082 15311 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.036971 15311 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.037303 15509 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
W20240503 07:59:52.037560 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.037680 17696 mvcc.cc:205] Tried to move back new op lower bound from 7023506194534866944 to 7023506194472730624. Current Snapshot: MvccSnapshot[applied={T|T < 7023506194534866944 or (T in {7023506194534866944})}]
I20240503 07:59:52.037806 17595 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.037984 17595 raft_consensus.cc:686] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.038100 17595 consensus_queue.cc:237] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 3.22, Last appended by leader: 22, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
W20240503 07:59:52.039830 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48464: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.043732 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.043838 15571 raft_consensus.cc:482] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.043898 15571 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.043869 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.044368 15705 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.045706 15571 raft_consensus.cc:504] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.046595 15571 leader_election.cc:288] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.046980 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.047084 15311 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.047762 15705 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.047752 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.047854 15311 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:52.047856 15442 raft_consensus.cc:3032] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:52.047910 15442 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.048025 15442 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 20, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.048132 15442 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.048319 15571 raft_consensus.cc:1226] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 2. Current term is 3. Ops: []
W20240503 07:59:52.048864 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.049047 15442 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:52.049124 15509 leader_election.cc:302] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.049261 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.049343 15574 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.049388 15574 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.049857 17595 raft_consensus.cc:2781] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:52.049937 17595 raft_consensus.cc:686] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.050047 17595 consensus_queue.cc:237] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.20, Last appended by leader: 20, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.050140 15574 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.050395 15574 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.051704 15574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434218808f44973b24e54e839a30e7c"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.051795 15574 raft_consensus.cc:482] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.051966 15574 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 2 FOLLOWER]: Advancing to term 3
W20240503 07:59:52.053146 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39004: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.053701 15574 raft_consensus.cc:504] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.053742 15704 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 07:59:52.053778 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.053864 15310 raft_consensus.cc:3032] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:52.053902 15310 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.054023 15310 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.054177 15310 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.054653 16990 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 07:59:52.054994 15310 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:52.055259 15509 leader_election.cc:302] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.055648 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.055747 15704 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
W20240503 07:59:52.055840 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.056078 17561 raft_consensus.cc:2781] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:52.056156 17561 raft_consensus.cc:686] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.056252 17561 consensus_queue.cc:237] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.056638 15704 raft_consensus.cc:2454] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:52.056278 15571 raft_consensus.cc:1261] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 07:59:52.057003 16990 consensus_queue.cc:1035] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:52.057157 15509 leader_election.cc:302] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.055284 15574 leader_election.cc:288] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 3 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.057364 17728 raft_consensus.cc:2781] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 07:59:52.057423 17728 raft_consensus.cc:686] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.057524 17728 consensus_queue.cc:237] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.061789 15312 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 22. Preceding OpId from leader: term: 4 index: 24. (index mismatch)
I20240503 07:59:52.061947 15703 raft_consensus.cc:1261] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 22. Preceding OpId from leader: term: 4 index: 24. (index mismatch)
I20240503 07:59:52.061805 15311 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 20. (index mismatch)
I20240503 07:59:52.062953 15704 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 20. (index mismatch)
I20240503 07:59:52.059602 15570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.063520 15570 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.063577 15570 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.063874 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48464: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.064441 15570 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.064991 15570 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.062745 17595 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 07:59:52.065297 17595 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240503 07:59:52.065510 17595 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20240503 07:59:52.066241 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:52.066347 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.067162 15163 catalog_manager.cc:5484] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.071338 17561 consensus_queue.cc:1035] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 07:59:52.072182 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.072283 15704 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.072467 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38988: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.072923 15703 raft_consensus.cc:1226] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 2. Current term is 3. Ops: []
I20240503 07:59:52.074118 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.074219 15442 raft_consensus.cc:3032] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.074261 15442 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.074404 15442 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
W20240503 07:59:52.074460 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54974: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.074551 15442 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.074882 16590 consensus_queue.cc:1046] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:52.075117 17720 raft_consensus.cc:3032] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 07:59:52.075162 17720 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.075264 17720 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.075392 17720 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.075762 15442 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.076686 15704 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.077281 15509 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.077477 17595 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.077551 17595 raft_consensus.cc:686] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.077656 17595 consensus_queue.cc:237] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.079959 15703 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 4 index: 21. (index mismatch)
I20240503 07:59:52.080335 17595 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 07:59:52.083519 15442 raft_consensus.cc:1261] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 4 index: 21. (index mismatch)
I20240503 07:59:52.083962 17595 consensus_queue.cc:1035] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20240503 07:59:52.088523 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.091234 15703 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 2 FOLLOWER]: Advancing to term 3
I20240503 07:59:52.089881 15310 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240503 07:59:52.092231 15703 raft_consensus.cc:1261] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
W20240503 07:59:52.092695 15404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.093394 17561 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:52.095242 15166 catalog_manager.cc:5484] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 2 to 3, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } }
I20240503 07:59:52.095955 17595 consensus_queue.cc:1035] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 07:59:52.097713 15570 raft_consensus.cc:1226] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 2. Current term is 3. Ops: []
I20240503 07:59:52.053754 15705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.101235 15705 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 3.
I20240503 07:59:52.106827 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.106976 15443 raft_consensus.cc:2396] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 33dfc6a1905a4bed9227e9e5b0cd9275 for term 3 because replica has last-logged OpId of term: 2 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
W20240503 07:59:52.112692 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.118307 15571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348"
dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
from {username='slave'} at 127.0.0.1:55742
I20240503 07:59:52.118464 15571 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.118561 15571 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.120182 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
W20240503 07:59:52.132014 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54988: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:52.135367 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38988: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.136617 15571 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
W20240503 07:59:52.137645 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.144901 15163 catalog_manager.cc:5484] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.145462 15163 catalog_manager.cc:5484] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.147444 15163 catalog_manager.cc:5484] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 2 to 3, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 3 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.148416 15310 raft_consensus.cc:1261] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
W20240503 07:59:52.154525 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.159137 15443 raft_consensus.cc:1261] T 83ec0f68443746bdbb34cb202a128e62 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20240503 07:59:52.148975 17595 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240503 07:59:52.159617 17595 consensus_queue.cc:1035] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20240503 07:59:52.159797 15571 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.160883 15703 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 07:59:52.162289 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.162398 15311 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.162948 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.163447 15311 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.165853 15509 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.166095 17728 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.166707 15313 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240503 07:59:52.166756 15313 pending_rounds.cc:76] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 4
I20240503 07:59:52.166777 15313 pending_rounds.cc:97] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240503 07:59:52.166800 15313 raft_consensus.cc:2866] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.167002 17728 raft_consensus.cc:686] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Leader. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.167141 17728 consensus_queue.cc:237] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.168423 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.168529 15443 raft_consensus.cc:3032] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.168565 15443 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.168699 15443 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.168866 15443 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.168934 17720 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:52.170871 15703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.171025 15703 raft_consensus.cc:482] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.171046 16590 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:52.171295 15703 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.172219 15703 raft_consensus.cc:504] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.172555 15703 leader_election.cc:288] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.174082 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.174171 15704 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.174211 15704 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.175149 15704 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.175457 15704 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.176095 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.176189 15310 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.177616 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.177713 15570 raft_consensus.cc:3032] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.177747 15570 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.177886 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.177901 15570 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.177979 15440 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.178021 15570 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.178982 15570 raft_consensus.cc:2396] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 07:59:52.178997 15440 raft_consensus.cc:2454] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.179235 15642 leader_election.cc:302] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.179358 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f31aec2eb34a41b3f725da97775274" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.179387 17542 raft_consensus.cc:2781] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.179448 15571 raft_consensus.cc:3032] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.179455 17542 raft_consensus.cc:686] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.179483 15571 raft_consensus.cc:729] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.179561 17542 consensus_queue.cc:237] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.179615 15571 consensus_queue.cc:260] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.179729 15571 raft_consensus.cc:3037] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.182163 15169 catalog_manager.cc:5484] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:52.182914 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.182976 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.183070 15704 raft_consensus.cc:482] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.183116 15704 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.184298 15310 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.184546 15640 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.180625 15571 raft_consensus.cc:2396] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 07:59:52.184697 16884 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.184823 16884 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.184932 16884 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.186319 17512 heartbeater.cc:503] Master 127.0.99.190:43357 was elected leader, sending a full tablet report...
I20240503 07:59:52.190414 15704 raft_consensus.cc:504] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.190945 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.191036 15571 raft_consensus.cc:3032] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.191069 15571 raft_consensus.cc:729] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.191233 15571 consensus_queue.cc:260] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 4.24, Last appended by leader: 24, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.191362 15571 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
W20240503 07:59:52.192090 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39022: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.195487 15571 raft_consensus.cc:2396] T 00d9d6231fc54c07ae77084649980ad1 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 22.
I20240503 07:59:52.195993 15704 leader_election.cc:288] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.198118 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.198216 15702 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.198266 15702 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.199111 15702 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.199419 15702 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.199702 15703 raft_consensus.cc:1226] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Rejecting Update request from peer 33dfc6a1905a4bed9227e9e5b0cd9275 for earlier term 4. Current term is 5. Ops: [4.23-4.23]
I20240503 07:59:52.200212 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.200369 15704 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.200476 15704 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.201267 15704 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.201627 15704 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.202219 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.202313 15440 raft_consensus.cc:3032] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.202351 15440 raft_consensus.cc:729] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.202463 15440 consensus_queue.cc:260] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 4.21, Last appended by leader: 21, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.202551 15440 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.202667 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.202752 15571 raft_consensus.cc:3032] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.202785 15571 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.202852 15169 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.202914 15571 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.11, Last appended by leader: 11, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.203029 15571 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.203577 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.203713 15704 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.203795 15571 raft_consensus.cc:2396] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240503 07:59:52.203827 15704 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.204653 15704 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.205046 15440 raft_consensus.cc:2396] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 20.
I20240503 07:59:52.205282 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.205396 15312 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.205479 15703 raft_consensus.cc:1226] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Rejecting Update request from peer 07ae48a2fef84a10a186e6256e28c064 for earlier term 4. Current term is 5. Ops: [4.21-4.21]
I20240503 07:59:52.205672 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.205803 15440 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.206321 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.206339 15312 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 07:59:52.207425 15440 raft_consensus.cc:2454] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.207703 15642 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters: c770157b083143399bcf339b4081a23f
I20240503 07:59:52.207881 17542 raft_consensus.cc:2781] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.207950 17542 raft_consensus.cc:686] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.208050 17542 consensus_queue.cc:237] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.208874 15640 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f
I20240503 07:59:52.209501 16884 raft_consensus.cc:2726] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240503 07:59:52.210275 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48442: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.210286 15704 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.211160 15169 catalog_manager.cc:5484] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 5, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.211695 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "420c787bccf84d02b8bf554193e72918"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.211787 15702 raft_consensus.cc:482] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.211834 15702 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.212590 15702 raft_consensus.cc:504] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.212983 15702 leader_election.cc:288] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.213182 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.213310 15440 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.214075 15440 raft_consensus.cc:2454] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.214351 15642 leader_election.cc:302] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.214408 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420c787bccf84d02b8bf554193e72918" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.214488 15571 raft_consensus.cc:3032] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.214521 15571 raft_consensus.cc:729] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.214524 16884 raft_consensus.cc:2781] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.214584 16884 raft_consensus.cc:686] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.214632 15571 consensus_queue.cc:260] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.214673 16884 consensus_queue.cc:237] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.214740 15571 raft_consensus.cc:3037] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.215495 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.215528 15571 raft_consensus.cc:2396] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 3 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11.
I20240503 07:59:52.215636 15704 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.215726 15704 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.217392 15169 catalog_manager.cc:5484] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 4, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 4 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:52.220135 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54998: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.220651 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.220757 15313 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.221717 15313 raft_consensus.cc:2454] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.221961 15640 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.222167 17542 raft_consensus.cc:2781] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.222247 17542 raft_consensus.cc:686] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.222357 17542 consensus_queue.cc:237] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.222524 15704 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.223088 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.223217 15440 raft_consensus.cc:3032] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.223294 15440 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.223287 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.223399 15312 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.224195 15312 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.224503 15640 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.224643 16884 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.224705 16884 raft_consensus.cc:686] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.224823 16884 consensus_queue.cc:237] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.224191 15169 catalog_manager.cc:5484] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.226649 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d9d6231fc54c07ae77084649980ad1" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.226747 15442 raft_consensus.cc:3037] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.230260 15443 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33dfc6a1905a4bed9227e9e5b0cd9275 in term 4.
I20240503 07:59:52.230487 15171 catalog_manager.cc:5484] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131). New cstate: current_term: 4 leader_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.230937 15442 raft_consensus.cc:2396] T 00d9d6231fc54c07ae77084649980ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 22.
I20240503 07:59:52.231375 15642 leader_election.cc:302] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.231542 17542 raft_consensus.cc:2726] T 00d9d6231fc54c07ae77084649980ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 07:59:52.223546 15440 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.234494 15440 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.235528 15169 catalog_manager.cc:5484] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 4 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.237401 15704 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.238106 15440 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.240716 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.240890 15704 raft_consensus.cc:482] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.240988 15704 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
W20240503 07:59:52.242269 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.243111 15704 raft_consensus.cc:504] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.244097 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.244256 15440 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.244994 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b399cf884c564b999fa07885e7bd6cb7" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.245114 15571 raft_consensus.cc:3032] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.245147 15571 raft_consensus.cc:729] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.245254 15571 consensus_queue.cc:260] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 4.10, Last appended by leader: 10, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.245393 15571 raft_consensus.cc:3037] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.246227 15571 raft_consensus.cc:2396] T b399cf884c564b999fa07885e7bd6cb7 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 4 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 9.
I20240503 07:59:52.247941 15704 leader_election.cc:288] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.248325 15440 raft_consensus.cc:2454] T b399cf884c564b999fa07885e7bd6cb7 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.248898 15703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.248992 15703 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.249039 15703 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.249053 15311 raft_consensus.cc:1261] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.249830 15703 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.250133 15703 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.249647 15442 raft_consensus.cc:1261] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.250886 16884 consensus_queue.cc:1035] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.250911 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.250982 16884 consensus_queue.cc:1225] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer c770157b083143399bcf339b4081a23f log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:52.250993 15312 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.251385 15642 leader_election.cc:302] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.251556 17542 consensus_queue.cc:1035] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.251793 17542 raft_consensus.cc:2781] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.251821 15312 raft_consensus.cc:2396] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 07:59:52.251852 17542 raft_consensus.cc:686] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.251961 17542 consensus_queue.cc:237] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.250480 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.252926 15443 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.253394 15169 catalog_manager.cc:5484] T b399cf884c564b999fa07885e7bd6cb7 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.253701 15443 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.254272 15642 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.254727 16884 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.254801 16884 raft_consensus.cc:686] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.254912 16884 consensus_queue.cc:237] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.258051 15311 pending_rounds.cc:76] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 3
I20240503 07:59:52.258096 15311 pending_rounds.cc:97] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 07:59:52.258126 15311 raft_consensus.cc:2866] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.258950 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.259047 15704 raft_consensus.cc:482] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.260953 15169 catalog_manager.cc:5484] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 5, leader changed from c770157b083143399bcf339b4081a23f (127.0.99.129) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.264561 15704 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.266467 15312 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 07:59:52.266860 17732 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 07:59:52.272274 15535 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55714: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.272567 17653 mvcc.cc:205] Tried to move back new op lower bound from 7023506195446104064 to 7023506191283019776. Current Snapshot: MvccSnapshot[applied={T|T < 7023506195446104064}]
W20240503 07:59:52.273514 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55004: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.278537 15440 raft_consensus.cc:1261] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 8. (index mismatch)
I20240503 07:59:52.279757 17561 consensus_queue.cc:1035] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:52.280133 15571 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.280740 17542 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:52.280859 17542 consensus_queue.cc:1225] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer 33dfc6a1905a4bed9227e9e5b0cd9275 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240503 07:59:52.281275 15313 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.281919 15571 pending_rounds.cc:76] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting all ops after (but not including) 4
I20240503 07:59:52.281970 15571 pending_rounds.cc:97] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240503 07:59:52.282018 15571 raft_consensus.cc:2866] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.282595 17542 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 07:59:52.305240 15704 raft_consensus.cc:504] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.305819 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.305932 15442 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.306887 15442 raft_consensus.cc:2396] T 964e76fe1aae447eb961f20965b1f754 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:52.307266 15704 leader_election.cc:288] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.308022 15442 raft_consensus.cc:1261] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.308320 17542 consensus_queue.cc:1035] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.308763 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964e76fe1aae447eb961f20965b1f754" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.308887 15571 raft_consensus.cc:3032] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.308931 15571 raft_consensus.cc:729] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.309049 15571 consensus_queue.cc:260] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.309166 15571 raft_consensus.cc:3037] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.310050 15571 raft_consensus.cc:2396] T 964e76fe1aae447eb961f20965b1f754 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 07:59:52.310645 15639 leader_election.cc:302] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.310896 16884 raft_consensus.cc:2726] T 964e76fe1aae447eb961f20965b1f754 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:52.315601 15311 raft_consensus.cc:1261] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.315889 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.315991 15704 raft_consensus.cc:482] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.316043 15704 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.317248 15704 raft_consensus.cc:504] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.318092 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.318182 15443 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.318208 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09079b0f21ed4bb0977c4d839e29d0dd" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.318310 15570 raft_consensus.cc:3037] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.319054 15570 raft_consensus.cc:2454] T 09079b0f21ed4bb0977c4d839e29d0dd P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.319082 15443 raft_consensus.cc:2396] T 09079b0f21ed4bb0977c4d839e29d0dd P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 6.
I20240503 07:59:52.319325 15639 leader_election.cc:302] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.319455 16884 consensus_queue.cc:1035] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.319531 16884 consensus_queue.cc:1225] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer c770157b083143399bcf339b4081a23f log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:52.319716 16884 raft_consensus.cc:2781] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.319775 16884 raft_consensus.cc:686] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.319887 16884 consensus_queue.cc:237] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.323295 15313 pending_rounds.cc:76] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 3
I20240503 07:59:52.323349 15313 pending_rounds.cc:97] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 07:59:52.323390 15313 raft_consensus.cc:2866] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.325224 15169 catalog_manager.cc:5484] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.336301 15443 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 07:59:52.337370 16884 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.339263 15312 raft_consensus.cc:1261] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 07:59:52.340899 17542 consensus_queue.cc:1035] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.342797 15312 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 4 index: 7. (term mismatch)
I20240503 07:59:52.342852 15312 pending_rounds.cc:76] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 6
I20240503 07:59:52.342875 15312 pending_rounds.cc:97] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f: Aborting uncommitted WRITE_OP operation due to leader change: 3.7
W20240503 07:59:52.342972 15312 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39026: Aborted: Op aborted by new leader
I20240503 07:59:52.344640 17584 mvcc.cc:205] Tried to move back new op lower bound from 7023506195806633984 to 7023506195354128384. Current Snapshot: MvccSnapshot[applied={T|T < 7023506195806633984 or (T in {7023506195806633984})}]
I20240503 07:59:52.345147 17719 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:52.349582 15702 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240503 07:59:52.353267 15703 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 07:59:52.353655 17729 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.354072 16990 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 07:59:52.361740 15310 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.365793 15313 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 07:59:52.366134 17729 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.368901 16990 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:52.372303 15702 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.372432 15704 leader_election.cc:288] T 09079b0f21ed4bb0977c4d839e29d0dd P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.373456 16990 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 07:59:52.389101 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55018: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.408208 15703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a95f1e683a94566927691769f5fda97"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.408334 15703 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.408390 15703 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.409370 15703 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.409816 15703 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.409950 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.410054 15570 raft_consensus.cc:3032] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.410092 15570 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.410218 15570 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.410372 15570 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.411283 15570 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.413470 15639 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.413858 16884 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.413938 16884 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.414049 16884 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
W20240503 07:59:52.420225 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55664: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.423244 15163 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.426667 15702 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 07:59:52.427004 16590 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:52.427876 15310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.427984 15310 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.429214 15310 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.432957 15310 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 07ae48a2fef84a10a186e6256e28c064: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 12. (term mismatch)
I20240503 07:59:52.433027 15310 pending_rounds.cc:76] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 11
I20240503 07:59:52.433245 15310 pending_rounds.cc:97] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f: Aborting uncommitted WRITE_OP operation due to leader change: 2.12
W20240503 07:59:52.433357 15310 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39012: Aborted: Op aborted by new leader
I20240503 07:59:52.433950 17720 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:52.438400 15312 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 14. (term mismatch)
I20240503 07:59:52.438460 15312 pending_rounds.cc:76] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Aborting all ops after (but not including) 13
I20240503 07:59:52.438481 15312 pending_rounds.cc:97] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f: Aborting uncommitted WRITE_OP operation due to leader change: 2.14
W20240503 07:59:52.438565 15312 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Aborted: Op aborted by new leader
I20240503 07:59:52.440083 15570 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20240503 07:59:52.440209 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48492: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.440510 17748 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 07:59:52.440946 17561 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:52.441893 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.441990 15702 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.442045 15702 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.446836 15702 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.447151 15702 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.448756 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.448887 15570 raft_consensus.cc:3032] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.448927 15570 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.449054 15570 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.449199 15570 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.450037 15570 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.450635 15639 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.450891 15310 raft_consensus.cc:1261] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 07:59:52.450966 17748 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.451023 17748 raft_consensus.cc:686] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.451133 17748 consensus_queue.cc:237] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.454914 17749 consensus_queue.cc:1035] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 07:59:52.456312 15702 raft_consensus.cc:1261] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 07:59:52.457805 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.457930 15702 raft_consensus.cc:482] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.458001 15702 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.458848 15702 raft_consensus.cc:504] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.459275 15702 leader_election.cc:288] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.459353 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.459455 15440 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.459960 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9579e24381dd4a909624688351db8f03"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.460072 15702 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.460418 15440 raft_consensus.cc:2396] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 07:59:52.460698 15702 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.462047 15702 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.462469 15702 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.463074 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.463164 15704 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.463212 15704 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.463997 15704 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.464259 15704 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.465616 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9434218808f44973b24e54e839a30e7c"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.465705 15704 raft_consensus.cc:482] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.465751 15704 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.466485 15704 raft_consensus.cc:504] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.466774 15704 leader_election.cc:288] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.466876 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973e048f97aa4fd19f8b38ef68a9ec84" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.466974 15571 raft_consensus.cc:3037] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.467314 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.467406 15574 raft_consensus.cc:3032] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.467437 15574 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.467548 15574 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.467681 15574 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.467837 15571 raft_consensus.cc:2454] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.468000 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.468068 15572 raft_consensus.cc:3032] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.468097 15572 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.468201 15572 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.468324 15572 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.468479 15574 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.469166 15572 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.469350 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.469452 15704 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.469499 15704 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.470208 15704 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
W20240503 07:59:52.470285 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55680: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.470512 15704 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.470759 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.470868 15443 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.470952 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.471022 15441 raft_consensus.cc:3032] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.471053 15441 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.471158 15441 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.471285 15441 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.471721 15443 raft_consensus.cc:2396] T 9434218808f44973b24e54e839a30e7c P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 2 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240503 07:59:52.472079 15441 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.472360 15642 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.473114 15639 leader_election.cc:302] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064
I20240503 07:59:52.474067 15166 catalog_manager.cc:5484] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 4, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 4 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
W20240503 07:59:52.474807 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54926: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.476619 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.477005 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9434218808f44973b24e54e839a30e7c" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.477092 15570 raft_consensus.cc:3032] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.477123 15570 raft_consensus.cc:729] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.477232 15570 consensus_queue.cc:260] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.477345 15570 raft_consensus.cc:3037] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.477715 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39032: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.478160 15570 raft_consensus.cc:2396] T 9434218808f44973b24e54e839a30e7c P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 3 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 11.
I20240503 07:59:52.478215 15309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.478304 15309 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.476810 17748 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.479000 17748 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.479104 17748 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 4.39, Last appended by leader: 39, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.479307 15309 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.476908 16884 raft_consensus.cc:2781] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.479821 16884 raft_consensus.cc:686] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.479938 16884 consensus_queue.cc:237] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.480037 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.480093 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.480130 15311 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.480161 15308 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
W20240503 07:59:52.477082 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48458: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.480895 15307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 39 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.480933 15704 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.480991 15704 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.480989 15307 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.481017 15311 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.478888 15639 leader_election.cc:302] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.481807 15307 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.482023 15639 leader_election.cc:302] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.482201 15308 raft_consensus.cc:2454] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.483390 17730 consensus_queue.cc:1035] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:52.484172 17683 mvcc.cc:205] Tried to move back new op lower bound from 7023506196230410240 to 7023506196129017856. Current Snapshot: MvccSnapshot[applied={T|T < 7023506196230410240}]
W20240503 07:59:52.485258 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38966: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.485663 15639 leader_election.cc:302] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 07ae48a2fef84a10a186e6256e28c064, 33dfc6a1905a4bed9227e9e5b0cd9275
I20240503 07:59:52.485801 16884 raft_consensus.cc:2781] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.486188 16884 raft_consensus.cc:686] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.486297 16884 consensus_queue.cc:237] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.486546 17542 raft_consensus.cc:2726] T 9434218808f44973b24e54e839a30e7c P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 07:59:52.482223 17748 raft_consensus.cc:2781] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.486991 17748 raft_consensus.cc:686] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.487104 17748 consensus_queue.cc:237] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.488160 15704 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.488714 15704 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.489559 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.489656 15308 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.489679 15163 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.489920 15163 catalog_manager.cc:5484] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.490118 15163 catalog_manager.cc:5484] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.490298 15163 catalog_manager.cc:5484] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 4, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 4 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.490358 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.490402 15308 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.490445 15442 raft_consensus.cc:3032] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.490480 15442 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.490633 15442 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.490778 15442 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.491683 15442 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
W20240503 07:59:52.494453 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54926: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
W20240503 07:59:52.500563 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54964: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.502707 15572 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 07:59:52.503046 15574 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240503 07:59:52.495007 15640 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.505295 15312 raft_consensus.cc:1261] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
W20240503 07:59:52.505780 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.506255 17782 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.506431 17782 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.506548 17782 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.503921 17542 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.509222 17542 consensus_queue.cc:1225] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer 33dfc6a1905a4bed9227e9e5b0cd9275 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240503 07:59:52.509478 15440 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20240503 07:59:52.510274 15442 raft_consensus.cc:1261] T c7f31aec2eb34a41b3f725da97775274 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240503 07:59:52.504034 17749 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:52.510748 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.510895 15702 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.510996 15702 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.511452 15310 raft_consensus.cc:1261] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 39. Preceding OpId from leader: term: 5 index: 41. (index mismatch)
I20240503 07:59:52.511492 15308 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.512104 15702 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.512192 17734 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:52.512636 15702 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), c770157b083143399bcf339b4081a23f (127.0.99.129:36571)
I20240503 07:59:52.516824 15163 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.517762 15570 pending_rounds.cc:76] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting all ops after (but not including) 5
I20240503 07:59:52.517812 15570 pending_rounds.cc:97] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 07:59:52.517838 15570 raft_consensus.cc:2866] T c7f31aec2eb34a41b3f725da97775274 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240503 07:59:52.521914 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54964: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.521978 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.522082 15443 raft_consensus.cc:3032] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.522121 15443 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.522258 15443 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.522431 15443 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.523419 15443 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.526211 17782 consensus_queue.cc:1035] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 07:59:52.528887 17782 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 07:59:52.529330 17782 consensus_queue.cc:1035] T c7f31aec2eb34a41b3f725da97775274 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 07:59:52.529834 15642 leader_election.cc:302] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.530107 17542 raft_consensus.cc:2781] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.530184 17542 raft_consensus.cc:686] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.530289 17542 consensus_queue.cc:237] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 4.8, Last appended by leader: 8, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
W20240503 07:59:52.531805 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.533190 15163 catalog_manager.cc:5484] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 4 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 5 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:52.538707 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55018: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.540990 17748 consensus_queue.cc:1035] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240503 07:59:52.541555 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.541643 15704 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.541692 15704 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.542625 15704 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.542932 15704 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.543983 15442 raft_consensus.cc:1261] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer 33dfc6a1905a4bed9227e9e5b0cd9275: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.544409 17734 consensus_queue.cc:1035] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:52.545939 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.546041 15570 raft_consensus.cc:3032] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.546078 15570 raft_consensus.cc:729] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.546192 15570 consensus_queue.cc:260] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.546343 15570 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.547271 15570 raft_consensus.cc:2396] T 110d0ab39d7c4f2aaae8967d49c90d09 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 3 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 15.
I20240503 07:59:52.548411 15572 raft_consensus.cc:1261] T 973e048f97aa4fd19f8b38ef68a9ec84 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.548796 17748 consensus_queue.cc:1035] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.549116 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 8 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.549232 15308 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.549602 15574 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.550204 15308 raft_consensus.cc:2454] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 5.
I20240503 07:59:52.550222 17748 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:52.550311 17748 consensus_queue.cc:1225] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer 33dfc6a1905a4bed9227e9e5b0cd9275 log is divergent from this leader: its last log entry 3.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20240503 07:59:52.551654 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.551720 15440 raft_consensus.cc:1261] T 420c787bccf84d02b8bf554193e72918 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20240503 07:59:52.551750 15308 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.551779 15444 raft_consensus.cc:1261] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 07:59:52.552524 15308 raft_consensus.cc:2396] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fed03e5364d24fb5b093f3a2007db5bf for term 4 because replica has last-logged OpId of term: 3 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 15.
I20240503 07:59:52.552739 15570 raft_consensus.cc:1203] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.10->[2.11-4.13] Dedup: 2.11->[4.12-4.13]
I20240503 07:59:52.552877 15570 pending_rounds.cc:76] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting all ops after (but not including) 11
I20240503 07:59:52.552906 15570 pending_rounds.cc:97] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275: Aborting uncommitted NO_OP operation due to leader change: 3.12
I20240503 07:59:52.552930 15570 raft_consensus.cc:2866] T 420c787bccf84d02b8bf554193e72918 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.553920 17749 consensus_queue.cc:1035] T 420c787bccf84d02b8bf554193e72918 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 07:59:52.553987 17783 consensus_queue.cc:1035] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 07:59:52.554113 17783 consensus_queue.cc:1225] T 973e048f97aa4fd19f8b38ef68a9ec84 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Peer 07ae48a2fef84a10a186e6256e28c064 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 07:59:52.554677 15441 pending_rounds.cc:76] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Aborting all ops after (but not including) 3
I20240503 07:59:52.554728 15441 pending_rounds.cc:97] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 07:59:52.554787 15441 raft_consensus.cc:2866] T 973e048f97aa4fd19f8b38ef68a9ec84 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 07:59:52.562193 15640 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fed03e5364d24fb5b093f3a2007db5bf; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f
I20240503 07:59:52.562851 17542 raft_consensus.cc:2726] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240503 07:59:52.565088 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.565778 15704 raft_consensus.cc:1226] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Rejecting Update request from peer 33dfc6a1905a4bed9227e9e5b0cd9275 for earlier term 3. Current term is 4. Ops: [3.16-3.16]
I20240503 07:59:52.571053 15704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc"
dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
from {username='slave'} at 127.0.0.1:48440
I20240503 07:59:52.571151 15704 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.571192 15704 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.572001 15704 raft_consensus.cc:504] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.572305 15704 leader_election.cc:288] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Requested vote from peers c770157b083143399bcf339b4081a23f (127.0.99.129:36571), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
W20240503 07:59:52.572829 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.573168 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.573261 15441 raft_consensus.cc:3032] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.573305 15441 raft_consensus.cc:729] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07ae48a2fef84a10a186e6256e28c064, State: Running, Role: LEADER
I20240503 07:59:52.573450 15441 consensus_queue.cc:260] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.573611 15441 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.574470 15441 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.577554 15642 leader_election.cc:302] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.577818 17784 raft_consensus.cc:2781] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Leader election won for term 4
I20240503 07:59:52.577956 17784 raft_consensus.cc:686] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Leader. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.578120 17784 consensus_queue.cc:237] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.581166 15312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc" candidate_uuid: "fed03e5364d24fb5b093f3a2007db5bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: true dest_uuid: "c770157b083143399bcf339b4081a23f"
I20240503 07:59:52.581274 15312 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.582172 15312 raft_consensus.cc:2454] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed03e5364d24fb5b093f3a2007db5bf in term 4.
I20240503 07:59:52.582726 15163 catalog_manager.cc:5484] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf reported cstate change: term changed from 3 to 4, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132). New cstate: current_term: 4 leader_uuid: "fed03e5364d24fb5b093f3a2007db5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
W20240503 07:59:52.586302 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48488: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.590010 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.590131 15312 raft_consensus.cc:482] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.590193 15312 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
W20240503 07:59:52.596743 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
W20240503 07:59:52.598244 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55018: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.597829 15312 raft_consensus.cc:504] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.599036 15312 leader_election.cc:288] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.599467 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.599831 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.599671 15570 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.600601 15312 raft_consensus.cc:482] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.600677 15312 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.601296 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3b87677104bd8b86e39f4bd81fdad" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 10 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.601413 15704 raft_consensus.cc:3032] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.601451 15704 raft_consensus.cc:729] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.601526 15312 raft_consensus.cc:504] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.601554 15704 consensus_queue.cc:260] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.601693 15704 raft_consensus.cc:3037] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.602089 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.602186 15444 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
W20240503 07:59:52.602617 15665 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48494: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.602957 15309 raft_consensus.cc:1261] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 5 index: 11. (index mismatch)
I20240503 07:59:52.603104 15444 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.603372 15249 leader_election.cc:302] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.602629 15704 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.603507 17662 raft_consensus.cc:2781] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.603574 17662 raft_consensus.cc:686] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.603689 17662 consensus_queue.cc:237] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.604072 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9f43270b7418d8c19a0cb5f4ec348" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 13 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.604163 15574 raft_consensus.cc:3032] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.604199 15574 raft_consensus.cc:729] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.604319 15574 consensus_queue.cc:260] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 4.13, Last appended by leader: 13, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.603693 15247 leader_election.cc:302] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.604467 15574 raft_consensus.cc:3037] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.604553 17656 raft_consensus.cc:2781] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 07:59:52.604611 17656 raft_consensus.cc:686] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.604709 17656 consensus_queue.cc:237] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.10, Last appended by leader: 10, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.605392 15574 raft_consensus.cc:2454] T c7e9f43270b7418d8c19a0cb5f4ec348 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.606212 15440 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
W20240503 07:59:52.607008 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.607662 15312 leader_election.cc:288] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.608237 15308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.608341 15308 raft_consensus.cc:482] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.608399 15308 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.608505 17784 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:52.609431 15163 catalog_manager.cc:5484] T c7e9f43270b7418d8c19a0cb5f4ec348 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.609943 15308 raft_consensus.cc:504] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.611606 15704 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
W20240503 07:59:52.611827 15403 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55032: Illegal state: replica 07ae48a2fef84a10a186e6256e28c064 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.611969 17662 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:52.612093 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.612183 15443 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.613189 15443 raft_consensus.cc:2454] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.615525 15570 raft_consensus.cc:2454] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.615531 15249 leader_election.cc:302] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.615680 17656 raft_consensus.cc:2781] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 07:59:52.615746 17656 raft_consensus.cc:686] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.615845 17656 consensus_queue.cc:237] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.615803 15163 catalog_manager.cc:5484] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 5 to 6, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 6 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.616748 15574 raft_consensus.cc:1261] T 77d3b87677104bd8b86e39f4bd81fdad P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 6 FOLLOWER]: Refusing update from remote peer c770157b083143399bcf339b4081a23f: Log matching property violated. Preceding OpId in replica: term: 5 index: 10. Preceding OpId from leader: term: 6 index: 12. (index mismatch)
W20240503 07:59:52.617475 15666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48488: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.618026 17662 consensus_queue.cc:1035] T 77d3b87677104bd8b86e39f4bd81fdad P c770157b083143399bcf339b4081a23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:52.618621 15310 raft_consensus.cc:1261] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 8. Preceding OpId from leader: term: 5 index: 10. (index mismatch)
I20240503 07:59:52.621891 15308 leader_election.cc:288] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.623011 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.623117 15309 raft_consensus.cc:482] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.623178 15309 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.624089 15309 raft_consensus.cc:504] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.624413 15309 leader_election.cc:288] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.624594 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09926775f4140d7a2a8bdb26abfe54a" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.624708 15704 raft_consensus.cc:3032] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.624752 15704 raft_consensus.cc:729] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.624887 15704 consensus_queue.cc:260] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.9, Last appended by leader: 9, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.625000 15704 raft_consensus.cc:3037] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.625489 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.626081 15704 raft_consensus.cc:2396] T f09926775f4140d7a2a8bdb26abfe54a P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20240503 07:59:52.626276 15313 raft_consensus.cc:482] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.626338 15313 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.627161 15313 raft_consensus.cc:504] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.627312 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.627405 15443 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.627449 15313 leader_election.cc:288] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.628744 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65f3357b1604254b1b15674013ee9d2" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 14 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.628867 15574 raft_consensus.cc:3032] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.628912 15574 raft_consensus.cc:729] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.629002 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 46 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.629030 15574 consensus_queue.cc:260] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 4.14, Last appended by leader: 14, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.629168 15444 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.629184 15574 raft_consensus.cc:3037] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.630048 15574 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.630187 15444 raft_consensus.cc:2454] T 06be4032feee4d269cc79a4969cf778f P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.630528 15313 raft_consensus.cc:1226] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Rejecting Update request from peer 33dfc6a1905a4bed9227e9e5b0cd9275 for earlier term 4. Current term is 5. Ops: []
I20240503 07:59:52.630645 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.630748 15309 raft_consensus.cc:482] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.630797 15309 raft_consensus.cc:3037] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.630811 15249 leader_election.cc:302] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
W20240503 07:59:52.630956 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.631137 17734 consensus_queue.cc:1046] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: INVALID_TERM, Last received: 4.46, Next index: 47, Last known committed idx: 45, Time since last communication: 0.000s
I20240503 07:59:52.631265 17734 raft_consensus.cc:3032] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.631300 17734 raft_consensus.cc:729] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.631415 17734 consensus_queue.cc:260] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 4.47, Last appended by leader: 47, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.631561 17734 raft_consensus.cc:3037] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.631585 15309 raft_consensus.cc:504] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.631990 15309 leader_election.cc:288] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.632229 17662 raft_consensus.cc:2781] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.632300 17662 raft_consensus.cc:686] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.632402 17662 consensus_queue.cc:237] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 4.46, Last appended by leader: 46, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.633718 15440 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240503 07:59:52.633960 15444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.634274 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.634402 15309 raft_consensus.cc:482] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.634474 15309 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.634708 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96a7a7715c3944c0a973563648a4c04e" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.634829 15704 raft_consensus.cc:2379] T 96a7a7715c3944c0a973563648a4c04e P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f in current term 5: Already voted for candidate fed03e5364d24fb5b093f3a2007db5bf in this term.
I20240503 07:59:52.635288 15309 raft_consensus.cc:504] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.617455 17784 consensus_queue.cc:1035] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 07:59:52.635790 15309 leader_election.cc:288] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.637151 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.637169 15444 raft_consensus.cc:2454] T 96a7a7715c3944c0a973563648a4c04e P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.637301 15704 raft_consensus.cc:3032] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.637347 15704 raft_consensus.cc:729] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.637461 15704 consensus_queue.cc:260] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 5.11, Last appended by leader: 11, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.637584 15704 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.637676 15249 leader_election.cc:302] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters: fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:52.637907 17656 raft_consensus.cc:2781] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.637974 17656 raft_consensus.cc:686] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.638068 17656 consensus_queue.cc:237] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 4.21, Last appended by leader: 21, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.636016 15442 raft_consensus.cc:1226] T f09926775f4140d7a2a8bdb26abfe54a P 07ae48a2fef84a10a186e6256e28c064 [term 6 FOLLOWER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 5. Current term is 6. Ops: [5.9-5.9]
I20240503 07:59:52.638726 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.638865 15309 raft_consensus.cc:482] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.639191 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06be4032feee4d269cc79a4969cf778f" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 46 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.639328 15574 raft_consensus.cc:2396] T 06be4032feee4d269cc79a4969cf778f P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 5 because replica has last-logged OpId of term: 4 index: 47, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 46.
I20240503 07:59:52.639602 15309 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.640059 15704 raft_consensus.cc:2396] T 78f510497d7442a9adff8c55290a9a9b P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
I20240503 07:59:52.640389 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f510497d7442a9adff8c55290a9a9b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.640480 15574 raft_consensus.cc:3037] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.641016 17781 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:52.641321 15574 raft_consensus.cc:2396] T 78f510497d7442a9adff8c55290a9a9b P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 4 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 10.
W20240503 07:59:52.642140 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.642624 15246 leader_election.cc:302] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.643028 17662 raft_consensus.cc:2781] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.643112 17662 raft_consensus.cc:686] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.643218 17662 consensus_queue.cc:237] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 4.14, Last appended by leader: 14, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.644212 15246 leader_election.cc:302] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c770157b083143399bcf339b4081a23f; no voters: 33dfc6a1905a4bed9227e9e5b0cd9275, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:52.645116 17656 raft_consensus.cc:2726] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 07:59:52.645776 15309 raft_consensus.cc:504] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.646288 15309 leader_election.cc:288] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.646335 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.646431 15441 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.646736 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1de740a3d10414d9d9f78279eea9200" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 13 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.646876 15704 raft_consensus.cc:3032] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.646952 15704 raft_consensus.cc:729] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.647010 15443 raft_consensus.cc:2454] T d65f3357b1604254b1b15674013ee9d2 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.647200 15704 consensus_queue.cc:260] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 5.14, Last appended by leader: 14, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.647296 15441 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P 07ae48a2fef84a10a186e6256e28c064 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 13.
I20240503 07:59:52.647706 15308 raft_consensus.cc:1226] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 5. Current term is 6. Ops: [5.14-5.14]
I20240503 07:59:52.648003 15313 raft_consensus.cc:1226] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 5. Current term is 6. Ops: [5.9-5.9]
I20240503 07:59:52.648334 15313 raft_consensus.cc:1261] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20240503 07:59:52.648691 15313 raft_consensus.cc:1226] T 78f510497d7442a9adff8c55290a9a9b P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Rejecting Update request from peer fed03e5364d24fb5b093f3a2007db5bf for earlier term 5. Current term is 6. Ops: [5.11-5.11]
I20240503 07:59:52.648800 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.648882 15309 raft_consensus.cc:482] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.648936 15309 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.649660 15309 raft_consensus.cc:504] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.649936 17781 consensus_queue.cc:1035] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 07:59:52.649967 15169 catalog_manager.cc:5484] T 06be4032feee4d269cc79a4969cf778f P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.650187 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.650224 15309 leader_election.cc:288] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.650278 15702 raft_consensus.cc:3032] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.650321 15702 raft_consensus.cc:729] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.650348 15169 catalog_manager.cc:5484] T d65f3357b1604254b1b15674013ee9d2 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.650434 15702 consensus_queue.cc:260] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.650588 15702 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.650555 15169 catalog_manager.cc:5484] T f09926775f4140d7a2a8bdb26abfe54a P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 5 to 6, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 6 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.650765 15169 catalog_manager.cc:5484] T 96a7a7715c3944c0a973563648a4c04e P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.651449 15702 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.651882 15247 leader_election.cc:302] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.652053 17656 raft_consensus.cc:2781] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.652132 17656 raft_consensus.cc:686] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.652242 17656 consensus_queue.cc:237] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.653158 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd9a8cfe24949debc36c83257263ad1" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.653255 15441 raft_consensus.cc:3037] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.653355 15309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a95f1e683a94566927691769f5fda97"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.653460 15309 raft_consensus.cc:482] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.653517 15309 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.653607 17759 consensus_queue.cc:1046] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: INVALID_TERM, Last received: 5.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240503 07:59:52.653934 15664 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48488: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.654052 15441 raft_consensus.cc:2454] T 9dd9a8cfe24949debc36c83257263ad1 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.654796 17542 consensus_queue.cc:1035] T 48f9f5f863324415ad1198a0c85ec9cc P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 07:59:52.654800 15309 raft_consensus.cc:504] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.655160 15704 raft_consensus.cc:3037] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.655483 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.655586 15702 raft_consensus.cc:3032] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.655629 15702 raft_consensus.cc:729] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.655748 15702 consensus_queue.cc:260] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.655889 15702 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.655952 15704 raft_consensus.cc:2396] T d1de740a3d10414d9d9f78279eea9200 P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 13.
I20240503 07:59:52.656486 15169 catalog_manager.cc:5484] T 9dd9a8cfe24949debc36c83257263ad1 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.656662 15702 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.656968 15247 leader_election.cc:302] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c770157b083143399bcf339b4081a23f; no voters: 07ae48a2fef84a10a186e6256e28c064, fed03e5364d24fb5b093f3a2007db5bf
I20240503 07:59:52.657174 17662 raft_consensus.cc:2726] T d1de740a3d10414d9d9f78279eea9200 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 07:59:52.658493 15247 leader_election.cc:302] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.658648 17662 raft_consensus.cc:2781] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 07:59:52.658725 17662 raft_consensus.cc:686] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.658833 17662 consensus_queue.cc:237] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.659603 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a95f1e683a94566927691769f5fda97" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.659708 15574 raft_consensus.cc:3037] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.660563 15574 raft_consensus.cc:2454] T 8a95f1e683a94566927691769f5fda97 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
W20240503 07:59:52.661757 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39012: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.663090 15309 leader_election.cc:288] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.663750 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.663882 15313 raft_consensus.cc:482] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.663930 15313 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.663846 15164 catalog_manager.cc:5484] T 8a95f1e683a94566927691769f5fda97 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 5 to 6, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 6 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } }
I20240503 07:59:52.664718 15313 raft_consensus.cc:504] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.665895 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.666009 15702 raft_consensus.cc:3032] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.666054 15702 raft_consensus.cc:729] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.666185 15702 consensus_queue.cc:260] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.666328 15702 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.667320 15702 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.667775 15247 leader_election.cc:302] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.668354 17662 raft_consensus.cc:2781] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.668442 17662 raft_consensus.cc:686] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.668550 17662 consensus_queue.cc:237] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
W20240503 07:59:52.670183 15534 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55728: Illegal state: replica 33dfc6a1905a4bed9227e9e5b0cd9275 is not leader of this config: current role FOLLOWER
I20240503 07:59:52.671487 15312 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 13. Preceding OpId from leader: term: 5 index: 15. (index mismatch)
I20240503 07:59:52.671813 17781 consensus_queue.cc:1035] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 07:59:52.673022 15164 catalog_manager.cc:5484] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.673910 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74cc2c49c6b94f50841c7edbb0598c33" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.674010 15574 raft_consensus.cc:3037] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.674787 15574 raft_consensus.cc:2454] T 74cc2c49c6b94f50841c7edbb0598c33 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.675827 15313 leader_election.cc:288] T 74cc2c49c6b94f50841c7edbb0598c33 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.677031 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.677135 15313 raft_consensus.cc:482] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.677192 15313 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.677950 15313 raft_consensus.cc:504] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.678244 15313 leader_election.cc:288] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.678579 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 26 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.678719 15443 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.678939 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.679042 15313 raft_consensus.cc:482] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.679096 15313 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.679731 15443 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.679745 15313 raft_consensus.cc:504] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.680033 15249 leader_election.cc:302] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ae48a2fef84a10a186e6256e28c064, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.680269 17656 raft_consensus.cc:2781] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.680279 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 44 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.680342 17656 raft_consensus.cc:686] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.680392 15702 raft_consensus.cc:3032] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.680424 15702 raft_consensus.cc:729] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.680444 17656 consensus_queue.cc:237] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 4.26, Last appended by leader: 26, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.680577 15702 consensus_queue.cc:260] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 5.44, Last appended by leader: 44, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.680717 15702 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.681635 15702 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.681720 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41a556e8bd54ad88a38ef933cf76443" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 26 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.681843 15574 raft_consensus.cc:3032] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.681885 15574 raft_consensus.cc:729] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.681923 15247 leader_election.cc:302] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.682011 15574 consensus_queue.cc:260] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 4.26, Last appended by leader: 26, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.682080 17662 raft_consensus.cc:2781] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 07:59:52.682153 15574 raft_consensus.cc:3037] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.682154 17662 raft_consensus.cc:686] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.682269 17662 consensus_queue.cc:237] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 5.44, Last appended by leader: 44, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.684301 15574 raft_consensus.cc:2454] T c41a556e8bd54ad88a38ef933cf76443 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
W20240503 07:59:52.685603 15271 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39018: Illegal state: replica c770157b083143399bcf339b4081a23f is not leader of this config: current role FOLLOWER
I20240503 07:59:52.686988 15313 leader_election.cc:288] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.687793 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b5e889c6b142f99e8facf2941e2983" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 44 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.687933 15443 raft_consensus.cc:3037] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.688807 15443 raft_consensus.cc:2454] T e5b5e889c6b142f99e8facf2941e2983 P 07ae48a2fef84a10a186e6256e28c064 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.689688 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.689833 15313 raft_consensus.cc:482] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.689926 15313 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.690181 15441 raft_consensus.cc:1261] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Refusing update from remote peer fed03e5364d24fb5b093f3a2007db5bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 13. Preceding OpId from leader: term: 5 index: 15. (index mismatch)
I20240503 07:59:52.691212 15313 raft_consensus.cc:504] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.691797 15313 leader_election.cc:288] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.692502 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.692636 15313 raft_consensus.cc:482] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.692730 15313 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.692824 15164 catalog_manager.cc:5484] T e5b5e889c6b142f99e8facf2941e2983 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 5 to 6, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 6 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.693035 15164 catalog_manager.cc:5484] T c41a556e8bd54ad88a38ef933cf76443 P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 4 to 5, leader changed from 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 5 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.693423 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.693511 15702 raft_consensus.cc:3032] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.693543 15702 raft_consensus.cc:729] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.693506 15313 raft_consensus.cc:504] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.693651 15702 consensus_queue.cc:260] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 5.16, Last appended by leader: 16, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.693740 15702 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.694067 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6e885ee89c4ac8836c944c8065af0b" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.694190 15442 raft_consensus.cc:3037] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.694475 15702 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.694741 15247 leader_election.cc:302] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.694907 17662 raft_consensus.cc:2781] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Leader election won for term 6
I20240503 07:59:52.694978 17662 raft_consensus.cc:686] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [term 6 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.695096 17662 consensus_queue.cc:237] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.695281 15443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"
I20240503 07:59:52.695367 15443 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.696144 15442 raft_consensus.cc:2454] T ec6e885ee89c4ac8836c944c8065af0b P 07ae48a2fef84a10a186e6256e28c064 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.698395 15313 leader_election.cc:288] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.699306 15312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9579e24381dd4a909624688351db8f03"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.699390 15312 raft_consensus.cc:482] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.699424 15312 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.700222 15312 raft_consensus.cc:504] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.700546 15312 leader_election.cc:288] T 9579e24381dd4a909624688351db8f03 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.702394 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70d8ae824d0d4f0796c23ba15fed9507" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.702560 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
W20240503 07:59:52.702617 15664 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48488: Illegal state: replica fed03e5364d24fb5b093f3a2007db5bf is not leader of this config: current role FOLLOWER
I20240503 07:59:52.702658 15313 raft_consensus.cc:482] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.702701 15313 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.702903 15570 raft_consensus.cc:3032] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.702975 15570 raft_consensus.cc:729] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.703148 15570 consensus_queue.cc:260] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 11, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.703325 15570 raft_consensus.cc:3037] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.703380 15313 raft_consensus.cc:504] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.703402 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.703496 15702 raft_consensus.cc:3032] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.703541 15702 raft_consensus.cc:729] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.704381 15313 leader_election.cc:288] T 83ec0f68443746bdbb34cb202a128e62 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 4 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077)
I20240503 07:59:52.704562 15570 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.702814 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9579e24381dd4a909624688351db8f03" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.704877 15574 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.704907 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.705008 15313 raft_consensus.cc:482] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.705086 15313 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.705772 15574 raft_consensus.cc:2454] T 9579e24381dd4a909624688351db8f03 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 6.
I20240503 07:59:52.705789 15313 raft_consensus.cc:504] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.705827 15702 consensus_queue.cc:260] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.31, Last appended by leader: 31, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.705953 15702 raft_consensus.cc:3037] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.706102 15313 leader_election.cc:288] T 263f44adeac1445bb1c38948ceb4b203 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.706387 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263f44adeac1445bb1c38948ceb4b203" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.706473 15704 raft_consensus.cc:3032] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Stepping down as leader of term 4
I20240503 07:59:52.706503 15704 raft_consensus.cc:729] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.706588 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.706597 15704 consensus_queue.cc:260] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.19, Last appended by leader: 19, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.706674 15313 raft_consensus.cc:482] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.706696 15704 raft_consensus.cc:3037] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.706720 15313 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.706933 15702 raft_consensus.cc:2396] T 9579e24381dd4a909624688351db8f03 P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 31, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 30.
I20240503 07:59:52.707374 15313 raft_consensus.cc:504] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.707535 15704 raft_consensus.cc:2396] T 263f44adeac1445bb1c38948ceb4b203 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 5 because replica has last-logged OpId of term: 4 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 18.
I20240503 07:59:52.708026 15313 leader_election.cc:288] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 6 election: Requested vote from peers 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.708312 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83ee2fb1964aa4b7c662377ab4b0e5" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.708425 15704 raft_consensus.cc:3032] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Stepping down as leader of term 5
I20240503 07:59:52.708458 15704 raft_consensus.cc:729] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fed03e5364d24fb5b093f3a2007db5bf, State: Running, Role: LEADER
I20240503 07:59:52.708549 15704 consensus_queue.cc:260] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.13, Last appended by leader: 13, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } } peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } }
I20240503 07:59:52.708575 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.708622 15704 raft_consensus.cc:3037] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Advancing to term 6
I20240503 07:59:52.708662 15313 raft_consensus.cc:482] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.708707 15313 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.709373 15704 raft_consensus.cc:2396] T 3d83ee2fb1964aa4b7c662377ab4b0e5 P fed03e5364d24fb5b093f3a2007db5bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c770157b083143399bcf339b4081a23f for term 6 because replica has last-logged OpId of term: 5 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 12.
I20240503 07:59:52.709450 15313 raft_consensus.cc:504] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.709808 15313 leader_election.cc:288] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132:41787)
I20240503 07:59:52.710577 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "468e137cf48f44488189628cd8aee5e0"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.710671 15313 raft_consensus.cc:482] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.710717 15313 raft_consensus.cc:3037] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.710618 15169 catalog_manager.cc:5484] T ec6e885ee89c4ac8836c944c8065af0b P c770157b083143399bcf339b4081a23f reported cstate change: term changed from 5 to 6, leader changed from fed03e5364d24fb5b093f3a2007db5bf (127.0.99.132) to c770157b083143399bcf339b4081a23f (127.0.99.129). New cstate: current_term: 6 leader_uuid: "c770157b083143399bcf339b4081a23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20240503 07:59:52.711621 15704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110d0ab39d7c4f2aaae8967d49c90d09" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 5 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: true dest_uuid: "fed03e5364d24fb5b093f3a2007db5bf"
I20240503 07:59:52.711701 15704 raft_consensus.cc:3037] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.712352 15704 raft_consensus.cc:2454] T 110d0ab39d7c4f2aaae8967d49c90d09 P fed03e5364d24fb5b093f3a2007db5bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.712630 15247 leader_election.cc:302] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c770157b083143399bcf339b4081a23f, fed03e5364d24fb5b093f3a2007db5bf; no voters:
I20240503 07:59:52.712831 17662 raft_consensus.cc:2781] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.712944 17662 raft_consensus.cc:686] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [term 5 LEADER]: Becoming Leader. State: Replica: c770157b083143399bcf339b4081a23f, State: Running, Role: LEADER
I20240503 07:59:52.713063 17662 consensus_queue.cc:237] T 110d0ab39d7c4f2aaae8967d49c90d09 P c770157b083143399bcf339b4081a23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "fed03e5364d24fb5b093f3a2007db5bf" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 41787 } }
I20240503 07:59:52.713407 15574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: true dest_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275"
I20240503 07:59:52.713533 15574 raft_consensus.cc:3032] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 07:59:52.713604 15574 raft_consensus.cc:729] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 33dfc6a1905a4bed9227e9e5b0cd9275, State: Running, Role: LEADER
I20240503 07:59:52.712826 15313 raft_consensus.cc:504] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } }
I20240503 07:59:52.714020 15574 consensus_queue.cc:260] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 3.22, Last appended by leader: 22, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c770157b083143399bcf339b4081a23f" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 36571 } } peers { permanent_uuid: "07ae48a2fef84a10a186e6256e28c064" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 36961 } } peers { permanent_uuid: "33dfc6a1905a4bed9227e9e5b0cd9275" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 35077 } }
I20240503 07:59:52.714108 15313 leader_election.cc:288] T 468e137cf48f44488189628cd8aee5e0 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Requested vote from peers 33dfc6a1905a4bed9227e9e5b0cd9275 (127.0.99.131:35077), 07ae48a2fef84a10a186e6256e28c064 (127.0.99.130:36961)
I20240503 07:59:52.714249 15574 raft_consensus.cc:3037] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 3 FOLLOWER]: Advancing to term 4
I20240503 07:59:52.714721 15313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48f9f5f863324415ad1198a0c85ec9cc"
dest_uuid: "c770157b083143399bcf339b4081a23f"
from {username='slave'} at 127.0.0.1:38982
I20240503 07:59:52.714818 15313 raft_consensus.cc:482] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 07:59:52.714877 15313 raft_consensus.cc:3037] T 48f9f5f863324415ad1198a0c85ec9cc P c770157b083143399bcf339b4081a23f [term 4 FOLLOWER]: Advancing to term 5
I20240503 07:59:52.715152 15574 raft_consensus.cc:2454] T 83ec0f68443746bdbb34cb202a128e62 P 33dfc6a1905a4bed9227e9e5b0cd9275 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 4.
I20240503 07:59:52.715446 15246 leader_election.cc:302] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33dfc6a1905a4bed9227e9e5b0cd9275, c770157b083143399bcf339b4081a23f; no voters:
I20240503 07:59:52.696300 15443 raft_consensus.cc:2454] T 70d8ae824d0d4f0796c23ba15fed9507 P 07ae48a2fef84a10a186e6256e28c064 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c770157b083143399bcf339b4081a23f in term 5.
I20240503 07:59:52.715646 17656 raft_consensus.cc:2781] T 70d8ae824d0d4f0796c23ba15fed9507 P c770157b083143399bcf339b4081a23f [term 5 FOLLOWER]: Leader election won for term 5
I20240503 07:59:52.703905 15440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ec0f68443746bdbb34cb202a128e62" candidate_uuid: "c770157b083143399bcf339b4081a23f" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: true dest_uuid: "07ae48a2fef84a10a186e6256e28c064"